Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 1.573855686s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:16:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.573883867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..................X......X..X.....X..............] duration: 2.943902007s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:16:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.944046538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..................X......X..X.....X.............] duration: 2.97446667s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:15:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.974491016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 1.63830193s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:15:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.638323668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..................X......X..X.....X............] duration: 2.913436866s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:14:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.913461099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 1.632925186s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:14:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.633000438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..................X......X..X.....X...........] duration: 3.0616652s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:14:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.061692789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 1.678720622s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:13:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.678742075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 2.172820305s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:13:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.172843603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X..................X......X..X.....X..........] duration: 2.94458297s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:13:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.944617656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.492508107s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:13:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492532205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X..................X......X..X.....X.........] duration: 3.612824008s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:13:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.612898601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.....................................................X.....] duration: 2.130298993s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:13:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.130333178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.151662445s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:12:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.151736007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X..................X......X..X.....X........] duration: 3.651437594s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:12:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.651487117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.015567948s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:11:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.015600381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.....................................................X....] duration: 1.710432038s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:11:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.710508626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.817796061s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.817831122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.929141911s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:11:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.929160735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X..................X......X..X.....X.......] duration: 3.106141964s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.106171699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X.....................................................X...] duration: 1.676235509s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:10:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.676324798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.865994151s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:10:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.866018551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X..................X......X..X.....X......] duration: 3.025627355s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.025652319s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.797432577s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:10:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.797456975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X.....................................................X..] duration: 1.705054658s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:09:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.705071714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.....................................................X.] duration: 2.158659102s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:09:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.158714553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X..................X......X..X.....X.....] duration: 3.161752581s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.16177764s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.859325104s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:09:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.859351073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.....................................................X] duration: 5.000884288s trace: https://jaeger.dev.storj.io/trace/5ca3bc2ac612f70f FAIL (2026-03-10 19:08:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000964567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.322715872s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.322776208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.271073489s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:08:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.271094759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.659082384s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.659153376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X..................X......X..X.....X....] duration: 3.083988346s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.084025744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.8482387s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:08:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848265512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.630222718s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:07:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.630252274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 2.204462755s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:07:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.204486472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..................X......X..X.....X...] duration: 3.062692787s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.062721117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 3.231585345s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.231609799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 2.029354868s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:07:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.029377698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.251358392s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:06:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.251383691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X..................X......X..X.....X..] duration: 3.100799649s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:06:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.100831185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.584128064s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.584152879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.945642904s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:06:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.945669777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.709769435s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.709869511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..................X......X..X.....X.] duration: 3.327604375s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:05:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.327622439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X..................X......X..X.....X] duration: 1m0.10456846s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.104594649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.866573747s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.866600507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.644758168s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:05:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.644786251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.704538953s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:04:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.704564224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X..................X......X..X.....] duration: 3.248105225s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:04:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.248130566s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)