Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X..........X........................] duration: 1.578682887s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:34:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.578707944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X......X..X.....X.......................................] duration: 3.019618484s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:34:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.019728682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X..........X.......................] duration: 2.14938803s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:34:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.149412522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X..........X......................] duration: 2.171089037s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:33:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.171119612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X......X..X.....X......................................] duration: 3.05082875s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:33:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.050855255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X......X..X.....X.....................................] duration: 3.049866574s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.049898157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X..........X.....................] duration: 1.675644852s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:32:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.675674133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X......X..X.....X....................................] duration: 3.047761729s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:32:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.047785931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X..........X....................] duration: 2.052974328s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:32:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.053004441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X......X..X.....X...................................] duration: 3.081923618s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.081955427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X..........X...................] duration: 1.663903005s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:31:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.663924508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X......X..X.....X..................................] duration: 2.856061229s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:31:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.856085592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X......X..X.....X.................................] duration: 3.141036965s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:30:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.141060577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X..........X..................] duration: 1.554737563s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:30:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.554872838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X..........X.................] duration: 1.67515017s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:30:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.675336337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X......X..X.....X................................] duration: 3.104447152s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:29:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.104472331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X..........X................] duration: 1.676698721s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:29:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.676724051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X......X..X.....X...............................] duration: 3.089612702s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:29:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.089636551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X......X..X.....X..............................] duration: 3.132875561s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:28:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.132910172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X..........X...............] duration: 1.689974842s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:28:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.690003721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X......X..X.....X.............................] duration: 3.269535751s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:28:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.269556395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X..........X..............] duration: 1.62099067s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:27:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.621015648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X......X..X.....X............................] duration: 3.011828997s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:27:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.011855605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X..........X.............] duration: 1.649439073s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:27:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.649488916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X......X..X.....X...........................] duration: 3.351394397s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:26:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.351419616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X..........X............] duration: 1.739068281s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:26:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.739091386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X..........X...........] duration: 1.744422934s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:26:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.744467496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X......X..X.....X..........................] duration: 3.493205274s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:25:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.493229388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X......X..X.....X.........................] duration: 3.130216562s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:25:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.130245229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X..........X..........] duration: 2.123549632s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:25:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.123575677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..................X......X..X.....X........................] duration: 3.174719802s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:24:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.174750971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X..........X.........] duration: 1.691751892s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:24:44 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.691776754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..................X......X..X.....X.......................] duration: 3.266336231s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:23:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.266359228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X..........X........] duration: 2.097000368s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:23:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.09702515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..................X......X..X.....X......................] duration: 2.795117362s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:22:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.795143042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X..........X.......] duration: 1.710996162s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:22:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.711024046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..................X......X..X.....X.....................] duration: 3.270513656s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:22:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.27054495s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X..........X......] duration: 1.591493752s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:21:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.591529344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..................X......X..X.....X....................] duration: 3.045662513s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:21:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.045694101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X..........X.....] duration: 2.220811431s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:20:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.220835722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..................X......X..X.....X...................] duration: 3.217336953s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.21736595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X..........X....] duration: 1.807240281s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:20:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.807300399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..................X......X..X.....X..................] duration: 3.047274288s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:19:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.047299673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X..........X...] duration: 2.196573136s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:19:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.196599235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..................X......X..X.....X.................] duration: 3.053312005s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:19:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.053340655s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X..........X..] duration: 1.870917114s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:18:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.870970256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..................X......X..X.....X................] duration: 3.123292372s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:17:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.123313923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X..........X.] duration: 2.138292009s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:17:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.138308743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X..........X] duration: 5.00103665s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:17:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.00105655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..................X......X..X.....X...............] duration: 3.168154599s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:16:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.168183631s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)