Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 2.167636732s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:59:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.167671493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.658895702s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:59:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.658977365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 2.223315253s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:58:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.223524499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.679815364s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:57:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.679840883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.659665166s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:56:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.659696971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.634871957s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:55:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.63490304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X..........X................................................] duration: 2.15019095s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:54:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.150213532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X..........X...............................................] duration: 1.61205829s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:53:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.612146017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X..........X..............................................] duration: 1.726304273s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:52:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.726345017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X..........X.............................................] duration: 1.661643391s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:51:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.661667935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X..........X............................................] duration: 1.728470826s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:50:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.728497038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X..........X...........................................] duration: 1.62171251s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:50:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.621733953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X..........X..........................................] duration: 2.148208862s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:49:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.14823141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X..........X.........................................] duration: 1.676153045s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:49:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.676217583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X..........X........................................] duration: 1.664707373s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:48:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.664729793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.076426615s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.076519937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X..........X.......................................] duration: 2.173589346s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:47:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.173613971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X..........X......................................] duration: 2.199825918s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:47:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.199912169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.045524611s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:46:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.045550242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.089260666s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:46:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.089294511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X..........X.....................................] duration: 1.598343414s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:45:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.598370539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.014578149s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:45:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.014602647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X..........X....................................] duration: 1.638937895s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:44:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.638975188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.082614101s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:44:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.082649064s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X..........X...................................] duration: 2.114385849s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:44:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.114423711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.016039377s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:43:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.016067808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration: 3.130574189s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:43:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.130594036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X..........X..................................] duration: 1.721312153s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:43:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.72133603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....X....................................................] duration: 3.072864174s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:42:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.072894411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X..........X.................................] duration: 2.098122409s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:42:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.098144601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration: 3.033513763s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:41:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.033542057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X..........X................................] duration: 1.543823074s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:41:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.543858086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X..........X...............................] duration: 1.636664634s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:41:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.636687539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..X.....X..................................................] duration: 3.074345642s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:40:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.074371513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X..........X..............................] duration: 1.616175882s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:40:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.616197584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..X.....X.................................................] duration: 2.703971468s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:40:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.703999967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..X.....X................................................] duration: 3.022545142s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:39:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.022569219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X..........X.............................] duration: 1.687559304s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:39:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.687616914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..X.....X...............................................] duration: 3.27133981s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:39:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.271371832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X..........X............................] duration: 1.654538122s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:38:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.654567261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..X.....X..............................................] duration: 3.026264724s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:38:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.026290655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..X.....X.............................................] duration: 3.099673486s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:37:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.099698704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X..........X...........................] duration: 1.533245156s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:37:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.53326752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..X.....X............................................] duration: 3.198489735s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:37:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.198519302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X..X.....X...........................................] duration: 3.208919727s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:36:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.208949593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X..........X..........................] duration: 1.67996161s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:36:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.680061184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......X..X.....X..........................................] duration: 3.071720375s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:36:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.071751373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......X..X.....X.........................................] duration: 3.087519452s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:35:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.087549406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X..........X.........................] duration: 1.680969415s trace: https://jaeger.dev.storj.io/trace/1f8ea9f2f808fa0c FAIL (2026-03-10 19:35:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.681003436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X......X..X.....X........................................] duration: 3.194367193s trace: https://jaeger.dev.storj.io/trace/874a268541bb717 FAIL (2026-03-10 19:35:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ar 10 19:04:33.223847103 (2h53m36.239516712s) [upload=2785]: Starting upload Mar 10 19:04:33.223934651 (2h53m36.239604261s) [upload=2785]: Done starting upload Mar 10 19:04:33.224041146 (2h53m36.239710746s) [upload=2785]: Uploading segments... Mar 10 19:04:33.224355397 (2h53m36.240024998s) [upload=2785]: Got next segment. Inline: false Mar 10 19:04:33.224407583 (2h53m36.240077188s) [upload=2785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:04:33.224415030 (2h53m36.240084634s) [upload=2785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:04:33.335275006 (2h53m36.350944606s) [seg_pos={0 0}, upload=2785]: Begin upload segment... Mar 10 19:04:33.335343209 (2h53m36.351012796s) [seg_pos={0 0}, upload=2785]: Done begin upload segment. Mar 10 19:04:33.335387546 (2h53m36.351057165s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335400558 (2h53m36.351070164s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335454031 (2h53m36.351123623s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.335873260 (2h53m36.351542861s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336048326 (2h53m36.351717935s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336271837 (2h53m36.351941439s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336430621 (2h53m36.35210023s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.336857821 (2h53m36.352527529s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337006295 (2h53m36.352675899s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337364628 (2h53m36.353034237s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337530101 (2h53m36.35319971s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.337722352 (2h53m36.353391958s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338070331 (2h53m36.353739934s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338078172 (2h53m36.353747771s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338411518 (2h53m36.35408112s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338510989 (2h53m36.354180608s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338698119 (2h53m36.354367718s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.338990444 (2h53m36.354660047s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Uploading piece... Mar 10 19:04:33.900883916 (2h53m36.916553507s) [upload=2785]: Next returned nil segment Mar 10 19:04:33.900898629 (2h53m36.916568232s) [upload=2785]: Waiting for error group managing segments... Mar 10 19:04:34.049516074 (2h53m37.065185675s) [noise=true , address=45.91.116.12:4256 , piece_id=IC7QKGJZR255EGCTUFBZO2OZGBKUHGOCS3MVWCQ6ABI63LOQOTRQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.062812832 (2h53m37.078482432s) [noise=true , address=45.91.116.6:4008 , piece_id=BCLMZYECYSQC6UQBQZTYEJHEE7YWJKGYQKQBZSOHTAUOMPKSDQNA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.068126980 (2h53m37.083796584s) [noise=true , address=145.79.227.45:4251 , piece_id=PFRMBXC24NI25MJINEQ3QWYXJTOQ7IJRCU3F4SXHQUUB2BLKMJNQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071512488 (2h53m37.087182087s) [noise=true , address=45.91.116.10:4108 , piece_id=INH4VKUOLC35QT4ZXAYRCICGLBVPSH7FA5ORAMONDFPG7RXWKQHA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.071614441 (2h53m37.087284041s) [noise=true , address=145.79.227.12:4797 , piece_id=MK7KDNECEBEF7IT7AEA4KE7TYMVLZLW3N2HOFWDNYCRAMLS4E55A, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.080756945 (2h53m37.096426551s) [noise=true , address=145.79.227.52:4474 , piece_id=DFUP2Z6CZ2FTRJHH6VC73XQLNCZSPGYE62CDA5MRDDK4T7EHIDFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.083649018 (2h53m37.09931862s) [noise=true , address=79.127.219.46:4923 , piece_id=NM5TAHGB777FSY5YCDLUX3ZBNNJYTE364334JJ5KKRFEWGXM63OA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.104594541 (2h53m37.120264145s) [noise=true , address=79.127.219.45:4871 , piece_id=YB6UDEDUWGYQRL2ORLZ7H2EUTPTYUJ7TJFNC2HQ7PE6H63VMTUIQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.401885031 (2h53m37.417554636s) [noise=true , address=167.17.72.69:4724 , piece_id=2KAPSHU5L7U3HTAZYNQX3BXMZBLHJI56YJZSRGHK2ROPNW7IPDZQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.560887284 (2h53m37.576556896s) [noise=true , address=193.149.164.68:4067 , piece_id=HEE66TSPB2UU2GXVXHITOPK4Z6BLOJCVISKH3UEMCZUMDJK2GIZA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.575526166 (2h53m37.591195766s) [noise=true , address=193.149.164.71:4314 , piece_id=ZAGXPY3V64KOAVCYHYDHJEAS5LJ6Z554GGR3OY4NQ5L52PYB4R2Q, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.629513242 (2h53m37.645182843s) [noise=true , address=79.127.205.234:4179 , piece_id=6ECEOLGGEKROJPJ4U732K2O5TZT3ZXJRAD5NLLPL4MRU7I6SBSBQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:04:34.729277472 (2h53m37.744947076s) [noise=true , address=193.149.164.69:4080 , piece_id=AALR7TNHM3U6FUMDEW2BF2K3SI2DQFM7CG5DLFHB4CYVAOOLVXFQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: <nil> Mar 10 19:05:32.953010861 (2h54m35.968680471s) [noise=true , address=77.90.31.15:4715 , piece_id=B44C3BISKF7DLVXNR3XN4UOP7RIRED2HMTDHOQ6JWVN3TZSXNDWA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953134803 (2h54m35.968804415s) [noise=true , address=77.90.31.13:4631 , piece_id=KPVCTYRLGVK7FGY63F7VN44AVBR7LFRCFTA5Y2GJL6QWY3YKLLAQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953216306 (2h54m35.968885913s) [noise=true , address=77.90.31.10:4525 , piece_id=MMTHNTAQY2OU2XWMWJORSQPGVVW6TXZEYM7I5A2BLE2QLEAOXEFA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:05:32.953445950 (2h54m35.969115559s) [noise=true , address=77.90.31.11:4561 , piece_id=6KCQBKRKVR2FQUYRMTNODRCCORWKPMP73DGLVBRHFW6LW3OPBJOQ, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:05:32.953713451 (2h54m35.96938306s) [noise=true , address=77.90.31.19:4869 , piece_id=I3PWG7BSVI6LA3JF3RH5TVP7SUMFW2TQLWHUIATR3JEOJVSXD7IA, seg_pos={0 0}, upload=2785]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 139.59.144.22:35654->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:05:32.953755267 (2h54m35.969424859s) [seg_pos={0 0}, upload=2785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:05:33.055584795 (2h54m36.071254395s) [upload=2785]: Done uploading segments... Mar 10 19:05:33.055621835 (2h54m36.071291436s) [upload=2785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.194393339s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)