Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration: 2.680323128s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.680347815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration: 2.492310451s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.492333304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 2.811129499s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:17:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.81115326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................................XXXXXXXX....................] duration: 1.624043634s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.624071344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................................XXXXXXXX...................] duration: 1.577232922s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:16:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.577282597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration: 2.203938365s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.203986884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 2.625955817s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.625969363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................................XXXXXXXX..................] duration: 1.522365004s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:16:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.522393294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration: 2.143301746s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:15:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.143328783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................................XXXXXXXX.................] duration: 1.529563641s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:15:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.52960621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 3.740528288s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:14:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.740556814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 2.2803124s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:14:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.280431002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................................XXXXXXXX................] duration: 1.570437793s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:14:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.570460349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 2.852351701s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.852376053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 2.244916478s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.244952193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 2.224090248s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:13:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.22411813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....................................XXXXXXXX...............] duration: 1.577524142s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:13:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.577560118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration: 2.889798187s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.889846465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......................................XXXXXXXX..............] duration: 1.649847943s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:12:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649882056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 2.861627773s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:12:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.861664152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 2.318861166s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:12:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.318950398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......................................XXXXXXXX.............] duration: 2.064227427s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.064253005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration: 2.283910692s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:11:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.283936458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 2.632429144s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:11:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632458815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.623205816s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:11:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623242897s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........................................XXXXXXXX............] duration: 1.544426596s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.544459049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........................................XXXXXXXX...........] duration: 2.032832323s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:11:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.03286202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.599010188s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:10:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.599043389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 2.726475022s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:10:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.726500844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........................................XXXXXXXX..........] duration: 1.673236893s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:10:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.673260168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 3.296776292s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:10:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.29684863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.484451512s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:10:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.484485489s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 2.269940857s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.26997207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 2.295458837s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:09:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.295486921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 2.892071069s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:09:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892129138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 2.785961662s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:09:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.785973941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.798149369s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:09:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.79817903s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........................................XXXXXXXX.........] duration: 1.601749746s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:09:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.601778076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.674552704s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:08:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.674574196s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration: 2.268961557s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:08:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.268984205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.565046858s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.565073756s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............................................XXXXXXXX........] duration: 1.537610881s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:08:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.537640063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration: 2.780191778s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:08:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.780216869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.808803809s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.808826614s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............................................XXXXXXXX.......] duration: 1.633320281s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.633336213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration: 2.274860779s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:07:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.274886705s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration: 2.750220936s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.75024314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.484555771s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.484587781s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.585759205s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:07:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.585787973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration: 2.260002892s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.260031427s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)