Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............XXXXXXXX......................................] duration: 1.525124226s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:30:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.525156492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............XXXXXXXX.....................................] duration: 1.567020188s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:29:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.567077337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration: 2.144835083s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:29:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.14485779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................XXXXXXXX....................................] duration: 1.661006477s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:29:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.661031582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration: 2.849023624s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:28:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.849054622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................XXXXXXXX...................................] duration: 1.564875789s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:28:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.564902272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration: 2.285166231s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:28:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.285194829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................XXXXXXXX..................................] duration: 1.540869699s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:28:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.540963586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration: 2.824293911s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824387915s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration: 3.164459193s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.164485852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration: 2.318738345s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:27:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.318769705s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................XXXXXXXX.................................] duration: 1.668301918s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668324103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................XXXXXXXX................................] duration: 1.597768489s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:26:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597784786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration: 2.639509618s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.639532804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration: 2.22315147s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.223180981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration: 2.877855779s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:26:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.877880661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration: 2.821167045s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:25:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821195741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....................XXXXXXXX...............................] duration: 1.543798564s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:25:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.543828448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration: 2.287202476s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:25:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.287228146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......................XXXXXXXX..............................] duration: 1.557890973s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:25:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.557916557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration: 2.224835537s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:24:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.224934271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 2.655513596s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.655537195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......................XXXXXXXX.............................] duration: 1.482283719s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.48231375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration: 2.259257348s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:24:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.259276787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........................XXXXXXXX............................] duration: 1.619450501s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:24:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.619478735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration: 2.322552171s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:23:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.322577093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 2.824698072s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:23:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824729588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 2.36095278s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:23:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.36098143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration: 2.825233642s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:23:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.82526715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........................XXXXXXXX...........................] duration: 1.638630703s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:22:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.638654308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 2.217722573s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.217742491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........................XXXXXXXX..........................] duration: 1.516950629s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:22:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.51697754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration: 3.102648012s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:22:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.102669063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration: 2.829034723s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:21:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829058779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration: 2.165732524s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:21:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.165779005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration: 2.810244186s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:20:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810268808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration: 2.417984874s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:20:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.418014786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........................XXXXXXXX.........................] duration: 1.531210964s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.531239435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............................XXXXXXXX........................] duration: 1.721694138s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:20:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.721743023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............................XXXXXXXX.......................] duration: 1.641766626s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:20:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.641807888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration: 2.747201804s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:19:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.747229063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration: 2.297125046s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:19:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.297147036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............................XXXXXXXX......................] duration: 2.042432313s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:19:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.042452895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration: 3.433205595s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:19:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433228207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration: 2.231074377s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:19:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.231163226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration: 2.74567755s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.745701403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration: 2.721005883s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:18:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721036463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration: 2.693420126s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:18:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.693448746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............................XXXXXXXX.....................] duration: 1.648215698s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:18:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.648237249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration: 2.230423002s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:18:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.2304869s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)