Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration: 2.909265061s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.909281288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration: 2.276468512s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:40:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.276528694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.XXXXXXXX...................................................] duration: 1.685721567s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:40:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685742259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..XXXXXXXX..................................................] duration: 1.544858247s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:39:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.544882304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration: 3.268252882s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:39:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.268279203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration: 2.253850533s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:39:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.253882428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration: 2.242027495s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:39:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.242058996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration: 2.952572126s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:38:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.952670786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...XXXXXXXX.................................................] duration: 1.543941781s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.543969717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration: 2.243178249s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:38:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.243205561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....XXXXXXXX................................................] duration: 1.642660362s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:37:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.642689386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration: 2.765209538s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:37:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.765233966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....XXXXXXXX...............................................] duration: 2.095161413s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.095183801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration: 2.228278532s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.228307657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration: 2.179822592s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:36:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.179858216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration: 2.774993293s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:36:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.77501628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......XXXXXXXX..............................................] duration: 1.671301169s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:36:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.671329324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration: 2.367229952s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:35:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.367250442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration: 2.85472446s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:35:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.854751142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration: 2.849183925s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:35:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.849203978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......XXXXXXXX.............................................] duration: 1.530178862s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:35:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.530204469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration: 2.370864577s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.37089754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration: 2.853443122s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:34:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.853484304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........XXXXXXXX............................................] duration: 1.557989869s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:34:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.55801609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration: 2.250792111s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:33:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.25082595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........XXXXXXXX...........................................] duration: 2.01964632s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:33:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.019680553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........XXXXXXXX..........................................] duration: 1.540778295s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:33:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.540809158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration: 2.781917198s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:33:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781960465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration: 2.15005623s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:32:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.150097942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........XXXXXXXX.........................................] duration: 1.539360613s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.539379283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration: 2.654642398s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:32:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.654671483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration: 2.251922554s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:32:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.251948629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration: 2.653095874s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.653134303s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............XXXXXXXX........................................] duration: 1.57283831s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572859349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration: 2.961377147s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.961393787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration: 2.13823124s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.138257347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............XXXXXXXX.......................................] duration: 1.55465101s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:30:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.554684969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration: 2.694232462s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:30:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.694258629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration: 2.11514638s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.115173035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration: 2.793216197s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.793237239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............XXXXXXXX......................................] duration: 1.525124226s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:30:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.525156492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............XXXXXXXX.....................................] duration: 1.567020188s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:29:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.567077337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration: 2.144835083s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:29:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.14485779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................XXXXXXXX....................................] duration: 1.661006477s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:29:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.661031582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration: 2.849023624s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:28:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.849054622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................XXXXXXXX...................................] duration: 1.564875789s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:28:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.564902272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration: 2.285166231s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:28:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.285194829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................XXXXXXXX..................................] duration: 1.540869699s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:28:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.540963586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration: 2.824293911s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824387915s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration: 3.164459193s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.164485852s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)