Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............................................XXXXXXXX......] duration: 1.575562455s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:06:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.575591077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration: 2.361916245s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:06:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.361934749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.025868723s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:06:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.025961683s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration: 2.781577599s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:06:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781598249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration: 3.01241167s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.012432896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............................................XXXXXXXX.....] duration: 1.670718551s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:06:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670743323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.490679057s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:05:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.490703701s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 2.247550958s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:05:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.247573828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 2.690206789s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:05:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.69030316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................................................XXXXXXXX....] duration: 1.7394347s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.739464066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.683288668s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:04:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.683310646s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................................................XXXXXXXX...] duration: 1.608127659s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:04:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.608150942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 2.295737826s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.29577949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 3.3414308s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:04:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.34145781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 2.796939271s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:04:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.796968578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.654208524s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:04:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.654236805s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................................................XXXXXXXX..] duration: 1.608615899s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:03:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.608637023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 2.358120406s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.358146986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 2.962602888s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:03:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.962637832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 2.871282133s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:03:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.871296542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................................................XXXXXXXX.] duration: 1.609045108s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.609068509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 52.039092401s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:03:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.039898854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 26.798525206s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:03:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 26.798555678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.698839708s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.698951115s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................................................XXXXXXXX] duration: 5.000641395s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:02:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000663199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.113842416s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:02:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.113861677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.449433417s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:02:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.449494174s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.001010955s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:02:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001024519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................................................XXXXXXX] duration: 5.000569077s trace: https://jaeger.dev.storj.io/trace/122ba807ddddffb2 FAIL (2025-11-30 10:02:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00078263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.68578433s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:01:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.68581983s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.108661384s trace: https://jaeger.dev.storj.io/trace/5628a655e2714d2e FAIL (2025-11-30 10:01:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== id=XOJRGBAEKHME5BQDEL66STXEG7PVGDI7IYMRAVL2ISXG7ICQVGUA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.555656058 (456h16m28.91431472s) [noise=true , address=45.91.116.13:4172 , piece_id=BHLF3Q54IVXSYU36WQM4S2J7TBB32I5ZSR6KENIEKVNNCJONV5EA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.556179468 (456h16m28.91483813s) [noise=true , address=145.79.227.2:4135 , piece_id=F4BCB5VMHELTOYNHVDBZ4M5LVATSSC4ZWPRICXORSL6CI5SWSGPQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.556642442 (456h16m28.915301096s) [noise=true , address=45.91.116.7:4016 , piece_id=X7YZPLT2SOWCX324UPT2MIN6GU2ELB4MW6SENWFDNUAJTVALK3JA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.556996827 (456h16m28.915655482s) [noise=true , address=77.90.31.19:4859 , piece_id=DNHLDGSC4XIDCAK6YO2BXEHVYG2I4G7HCGD4I5IKVVUXWWUIWQ3Q, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.558062997 (456h16m28.91672166s) [noise=true , address=77.90.31.10:4530 , piece_id=COV3YPGFFLMMOJJDM3NPO2D3BVH7UKFAAU3XAVXVTSSCOZGM2XDA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.558143407 (456h16m28.916802066s) [noise=true , address=64.49.13.169:4707 , piece_id=PRLZJOUTQV5DJCXJU6ZH3GY2QGXISID7BO3UMSMS4NAV2ZNROYZA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.558617257 (456h16m28.917275917s) [noise=true , address=77.90.31.14:4682 , piece_id=T6RDDTVITD4XQGR43FT6BFBEQGT5BNHUSIGZ4MIL7AEQ2FW3MAPQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.559762016 (456h16m28.918420675s) [noise=true , address=77.90.31.12:4624 , piece_id=HPPAFDZAM37GD667FSFBTPHWBM23EOF7ESLPL5PC33B6E4SEVSWA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.560710995 (456h16m28.919369657s) [noise=true , address=77.90.31.11:4584 , piece_id=52U3DVP2LNN3T6PGURUT2F5UIVFJOTCSCBU77JQIZ22AWLF2ASUA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.561343567 (456h16m28.920002229s) [noise=true , address=77.90.31.16:4759 , piece_id=U5LIMWBCKEZAT4X6UIWED7GODEBZMTPTZYRCQUWQZWAV7ZDEX4LA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.562419638 (456h16m28.9210783s) [noise=true , address=77.90.31.13:4647 , piece_id=Z5ZEXH742CUA4JOOCI5QSWXBMI3JAYCUFF5RUV42GV6FWKXOA4PQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.562929282 (456h16m28.921587944s) [noise=true , address=45.91.116.6:4000 , piece_id=KY47VYFQWOPWVXBGXAORHML7SELYUHWGROGFPWNNOGLGZTQOVKLQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.564000684 (456h16m28.922659348s) [noise=true , address=77.90.31.15:4712 , piece_id=7XYUEFMB6SUGSEXFQHLUGPZE67AFMHHGWI7LLZNHTKKXHEBOZNMQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.564309110 (456h16m28.922967768s) [noise=true , address=45.91.116.14:4197 , piece_id=Y5V6BFPDD6RT2N4WJEFJFNJRFZHNCX4QQ64OA3A7TD23SBC7XU2Q, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.565625482 (456h16m28.924284127s) [noise=true , address=45.91.116.8:4076 , piece_id=LHEM4HDNGODKJER7HLCAQKAK372WSFE6FANWHNTBUHLALHOCVZCA, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.567442416 (456h16m28.926101078s) [noise=true , address=45.91.116.11:4136 , piece_id=RRDALXZG3PFCPERJOZJZL2IK3V4WDUDIUE2LGCR5QRS4FAVO5BZQ, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:43.569123741 (456h16m28.927782403s) [noise=true , address=64.49.13.139:4474 , piece_id=JCVXHNVHHTXMQDHRQ4QCCDB4M6YEWCXZR2LVGLNZ42Z5LY3B767A, seg_pos={0 0}, upload=401871]: Uploading piece... Nov 30 10:00:44.125700403 (456h16m29.484359057s) [upload=401871]: Next returned nil segment Nov 30 10:00:44.125715039 (456h16m29.484373684s) [upload=401871]: Waiting for error group managing segments... Nov 30 10:00:44.263006322 (456h16m29.621664982s) [noise=true , address=45.91.116.5:4314 , piece_id=XOJRGBAEKHME5BQDEL66STXEG7PVGDI7IYMRAVL2ISXG7ICQVGUA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.263255544 (456h16m29.621914207s) [noise=true , address=45.91.116.8:4076 , piece_id=LHEM4HDNGODKJER7HLCAQKAK372WSFE6FANWHNTBUHLALHOCVZCA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.263480260 (456h16m29.622138926s) [noise=true , address=45.91.116.6:4000 , piece_id=KY47VYFQWOPWVXBGXAORHML7SELYUHWGROGFPWNNOGLGZTQOVKLQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.265495999 (456h16m29.624154663s) [noise=true , address=145.79.227.2:4135 , piece_id=F4BCB5VMHELTOYNHVDBZ4M5LVATSSC4ZWPRICXORSL6CI5SWSGPQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.267060690 (456h16m29.625719379s) [noise=true , address=45.91.116.13:4172 , piece_id=BHLF3Q54IVXSYU36WQM4S2J7TBB32I5ZSR6KENIEKVNNCJONV5EA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.277782834 (456h16m29.636441498s) [noise=true , address=45.91.116.14:4197 , piece_id=Y5V6BFPDD6RT2N4WJEFJFNJRFZHNCX4QQ64OA3A7TD23SBC7XU2Q, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.280279153 (456h16m29.63893782s) [noise=true , address=45.91.116.7:4016 , piece_id=X7YZPLT2SOWCX324UPT2MIN6GU2ELB4MW6SENWFDNUAJTVALK3JA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.285323481 (456h16m29.643982148s) [noise=true , address=45.91.116.11:4136 , piece_id=RRDALXZG3PFCPERJOZJZL2IK3V4WDUDIUE2LGCR5QRS4FAVO5BZQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:44.862070509 (456h16m30.220729162s) [noise=true , address=64.49.13.139:4474 , piece_id=JCVXHNVHHTXMQDHRQ4QCCDB4M6YEWCXZR2LVGLNZ42Z5LY3B767A, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:00:45.753726767 (456h16m31.112385424s) [noise=true , address=64.49.13.169:4707 , piece_id=PRLZJOUTQV5DJCXJU6ZH3GY2QGXISID7BO3UMSMS4NAV2ZNROYZA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: <nil> Nov 30 10:01:43.139142558 (456h17m28.497801226s) [noise=true , address=77.90.31.16:4759 , piece_id=U5LIMWBCKEZAT4X6UIWED7GODEBZMTPTZYRCQUWQZWAV7ZDEX4LA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139164164 (456h17m28.497822824s) [noise=true , address=77.90.31.10:4530 , piece_id=COV3YPGFFLMMOJJDM3NPO2D3BVH7UKFAAU3XAVXVTSSCOZGM2XDA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: write tcp 139.59.144.22:37202->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139433674 (456h17m28.498092348s) [noise=true , address=77.90.31.15:4712 , piece_id=7XYUEFMB6SUGSEXFQHLUGPZE67AFMHHGWI7LLZNHTKKXHEBOZNMQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139510287 (456h17m28.498168948s) [noise=true , address=77.90.31.14:4682 , piece_id=T6RDDTVITD4XQGR43FT6BFBEQGT5BNHUSIGZ4MIL7AEQ2FW3MAPQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d, address:77.90.31.14:4682): protocol: rpc: dial tcp 77.90.31.14:4682: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139650828 (456h17m28.498309514s) [noise=true , address=77.90.31.13:4647 , piece_id=Z5ZEXH742CUA4JOOCI5QSWXBMI3JAYCUFF5RUV42GV6FWKXOA4PQ, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: rpc: dial tcp 77.90.31.13:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139710617 (456h17m28.498369278s) [noise=true , address=77.90.31.12:4624 , piece_id=HPPAFDZAM37GD667FSFBTPHWBM23EOF7ESLPL5PC33B6E4SEVSWA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.139892669 (456h17m28.498551331s) [noise=true , address=77.90.31.11:4584 , piece_id=52U3DVP2LNN3T6PGURUT2F5UIVFJOTCSCBU77JQIZ22AWLF2ASUA, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.140086168 (456h17m28.498744824s) [noise=true , address=77.90.31.19:4859 , piece_id=DNHLDGSC4XIDCAK6YO2BXEHVYG2I4G7HCGD4I5IKVVUXWWUIWQ3Q, seg_pos={0 0}, upload=401871]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: rpc: dial tcp 77.90.31.19:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:43.140114408 (456h17m28.498773047s) [seg_pos={0 0}, upload=401871]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:01:43.246172876 (456h17m28.604831823s) [upload=401871]: Done uploading segments... Nov 30 10:01:43.246233058 (456h17m28.604891721s) [upload=401871]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.10867702s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................................................XXXXXX] duration: 5.000888603s trace: https://jaeger.dev.storj.io/trace/4e4e5e6ac207973c FAIL (2025-11-30 10:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000918448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000735814s trace: https://jaeger.dev.storj.io/trace/44466118db2fbc1f FAIL (2025-11-30 10:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00074958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.708060232s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:01:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.708089809s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.111935255s trace: https://jaeger.dev.storj.io/trace/4d12defe02c60bbd FAIL (2025-11-30 10:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (456h15m28.667113053s) [seg_pos={0 0}, upload=401854]: Begin upload segment... Nov 30 09:59:43.308513542 (456h15m28.667172189s) [seg_pos={0 0}, upload=401854]: Done begin upload segment. Nov 30 09:59:43.308553389 (456h15m28.667212044s) [noise=true , address=145.79.227.18:4019 , piece_id=PYV5CZERTOJLHRZJHB7N2SQGFPPODOGUHB7EZEWBBXM6L2YSAVNA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.308568383 (456h15m28.667227044s) [noise=true , address=45.91.116.9:4268 , piece_id=EP7SVL7JFPW7IW62TM2ZRNO7GO3KEH6VI3CLKA2V2AIQGRVL7UOA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.309200969 (456h15m28.667859622s) [noise=true , address=77.90.31.17:4795 , piece_id=PI25BKP6SZGFV4S4HLVEACU3WJO5GNXNFFYKXXOK3HOTSM4FLGFA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.309825002 (456h15m28.668483673s) [noise=true , address=45.91.116.12:4242 , piece_id=RQZ6B77TYSH72NJVHCNSDOJJEKGO3SXWEEH4NIBBAIENYU3UUUTQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.310582563 (456h15m28.66924122s) [noise=true , address=77.90.31.15:4716 , piece_id=KSJ5KUP5T5LNLQOFTFELYB5NY36JR657OOAI5Q4TVNZEA37WWCFQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.310705948 (456h15m28.669364612s) [noise=true , address=45.91.116.6:4007 , piece_id=PFZTHESUO6ITWLIJWP2M7VJJMWPSDF2KHSNAUBS7KYUWEEAIFA7A, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.311019527 (456h15m28.669678186s) [noise=true , address=45.91.116.4:4944 , piece_id=WDSCTP5DXWNZDVI56RKZ4FVLDCQR3P3BVU6YK4TWHTW4Y3WLMO6Q, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.311342901 (456h15m28.670001567s) [noise=true , address=77.90.31.21:4895 , piece_id=3INM3P3QVTL6YFKUDTAMGAWPVL4JVJ3UZOBVPNGITMY5VROMRNMA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.311485513 (456h15m28.670144177s) [noise=true , address=77.90.31.16:4762 , piece_id=ILR4J42DR45YKHJZNKZTFC3XRWKBVNH2IDI2XWBK4BKGGTKSDWIA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.311892670 (456h15m28.670551326s) [noise=true , address=77.90.31.19:4872 , piece_id=NG7MKTA36FOXPGLNGXE6PMYT2KUEN372QIKYPSDZR6NOIVBR3UEQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.311905824 (456h15m28.670564485s) [noise=true , address=77.90.31.18:4814 , piece_id=3H7JDBHBMA2J7F2MWLVV4H7IMD3UA6MUBI74QPBRFOBSJZVCXTIQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.312254126 (456h15m28.670912782s) [noise=true , address=145.79.227.8:4362 , piece_id=R3A3AEYUG26Q65EKRSX55IVD74D3BSZZXTXM35MOKOFSZ7WFE6TA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.312458767 (456h15m28.671117412s) [noise=true , address=45.91.116.10:4097 , piece_id=O7CS55T2RAHUNX43CUKAWB4AGISF5RIWBMUW7YL5K2CREXI65EYA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.312671215 (456h15m28.671329874s) [noise=true , address=45.91.116.5:4304 , piece_id=3IQMWJ6OIDSOFBUJ52AR4XZ4WATVGX655AKB5RFZ5RSMBZD24P2Q, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.312911644 (456h15m28.671570302s) [noise=true , address=45.91.116.13:4184 , piece_id=MY6CL2OLBQFWK6OYYBVK3STHEQGN7GHKI2G2EZVYHIZA434WV2EA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.313109705 (456h15m28.671768361s) [noise=true , address=45.91.116.7:4032 , piece_id=WF53VFEPFD5IMAMF7XJHA7O5Y35AXAS6VHRWZ6YEE6GFVY325UFA, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.313342809 (456h15m28.672001464s) [noise=true , address=64.49.13.169:4716 , piece_id=OM7PKRIKWEZ2GMFUXNZUKSFL7DD5OAWDLH577I6TSPC6DFKSFRJQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.313532651 (456h15m28.672191307s) [noise=true , address=64.49.13.154:4936 , piece_id=IB3SAVX4ELGYCTVQO3YAVVT3AAMMMLJDQWHCRCU7JTDVLYJ2GUTQ, seg_pos={0 0}, upload=401854]: Uploading piece... Nov 30 09:59:43.867747243 (456h15m29.226405897s) [upload=401854]: Next returned nil segment Nov 30 09:59:43.867760314 (456h15m29.226418968s) [upload=401854]: Waiting for error group managing segments... Nov 30 09:59:44.017778679 (456h15m29.376437332s) [noise=true , address=45.91.116.4:4944 , piece_id=WDSCTP5DXWNZDVI56RKZ4FVLDCQR3P3BVU6YK4TWHTW4Y3WLMO6Q, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.018172219 (456h15m29.376830862s) [noise=true , address=45.91.116.5:4304 , piece_id=3IQMWJ6OIDSOFBUJ52AR4XZ4WATVGX655AKB5RFZ5RSMBZD24P2Q, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.019228033 (456h15m29.377886689s) [noise=true , address=145.79.227.8:4362 , piece_id=R3A3AEYUG26Q65EKRSX55IVD74D3BSZZXTXM35MOKOFSZ7WFE6TA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.019496962 (456h15m29.378155616s) [noise=true , address=45.91.116.13:4184 , piece_id=MY6CL2OLBQFWK6OYYBVK3STHEQGN7GHKI2G2EZVYHIZA434WV2EA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.019699040 (456h15m29.378357682s) [noise=true , address=45.91.116.6:4007 , piece_id=PFZTHESUO6ITWLIJWP2M7VJJMWPSDF2KHSNAUBS7KYUWEEAIFA7A, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.021495487 (456h15m29.380154143s) [noise=true , address=145.79.227.18:4019 , piece_id=PYV5CZERTOJLHRZJHB7N2SQGFPPODOGUHB7EZEWBBXM6L2YSAVNA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.023134027 (456h15m29.38179268s) [noise=true , address=45.91.116.9:4268 , piece_id=EP7SVL7JFPW7IW62TM2ZRNO7GO3KEH6VI3CLKA2V2AIQGRVL7UOA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.041093057 (456h15m29.399751715s) [noise=true , address=45.91.116.10:4097 , piece_id=O7CS55T2RAHUNX43CUKAWB4AGISF5RIWBMUW7YL5K2CREXI65EYA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.046815142 (456h15m29.405473807s) [noise=true , address=45.91.116.7:4032 , piece_id=WF53VFEPFD5IMAMF7XJHA7O5Y35AXAS6VHRWZ6YEE6GFVY325UFA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.104543397 (456h15m29.46320205s) [noise=true , address=45.91.116.12:4242 , piece_id=RQZ6B77TYSH72NJVHCNSDOJJEKGO3SXWEEH4NIBBAIENYU3UUUTQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.552103556 (456h15m29.910762231s) [noise=true , address=64.49.13.154:4936 , piece_id=IB3SAVX4ELGYCTVQO3YAVVT3AAMMMLJDQWHCRCU7JTDVLYJ2GUTQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 09:59:44.559901738 (456h15m29.918560392s) [noise=true , address=64.49.13.169:4716 , piece_id=OM7PKRIKWEZ2GMFUXNZUKSFL7DD5OAWDLH577I6TSPC6DFKSFRJQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: <nil> Nov 30 10:00:42.896192623 (456h16m28.254851276s) [noise=true , address=77.90.31.17:4795 , piece_id=PI25BKP6SZGFV4S4HLVEACU3WJO5GNXNFFYKXXOK3HOTSM4FLGFA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 139.59.144.22:45578->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896459638 (456h16m28.255118291s) [noise=true , address=77.90.31.15:4716 , piece_id=KSJ5KUP5T5LNLQOFTFELYB5NY36JR657OOAI5Q4TVNZEA37WWCFQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:12tuEGhEzrpo1eV7v2QatHXHMy7b1S5gcVSymWYWJSoCFUMGnaL, address:77.90.31.15:4716): protocol: write tcp 139.59.144.22:53736->77.90.31.15:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896534310 (456h16m28.255192966s) [noise=true , address=77.90.31.21:4895 , piece_id=3INM3P3QVTL6YFKUDTAMGAWPVL4JVJ3UZOBVPNGITMY5VROMRNMA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896617944 (456h16m28.255276603s) [noise=true , address=77.90.31.16:4762 , piece_id=ILR4J42DR45YKHJZNKZTFC3XRWKBVNH2IDI2XWBK4BKGGTKSDWIA, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: write tcp 139.59.144.22:33284->77.90.31.16:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896762261 (456h16m28.255420927s) [noise=true , address=77.90.31.18:4814 , piece_id=3H7JDBHBMA2J7F2MWLVV4H7IMD3UA6MUBI74QPBRFOBSJZVCXTIQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 139.59.144.22:52746->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896867750 (456h16m28.255526407s) [noise=true , address=77.90.31.19:4872 , piece_id=NG7MKTA36FOXPGLNGXE6PMYT2KUEN372QIKYPSDZR6NOIVBR3UEQ, seg_pos={0 0}, upload=401854]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 139.59.144.22:46474->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:42.896901673 (456h16m28.255560318s) [seg_pos={0 0}, upload=401854]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 10:00:43.005566405 (456h16m28.364225094s) [upload=401854]: Done uploading segments... Nov 30 10:00:43.005609932 (456h16m28.36426861s) [upload=401854]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.111952912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.556919294s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:00:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.556940775s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......................................................XXXXX] duration: 5.002041205s trace: https://jaeger.dev.storj.io/trace/c24965893106acc FAIL (2025-11-30 10:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002068503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001190563s trace: https://jaeger.dev.storj.io/trace/5073e07c70ffc650 FAIL (2025-11-30 10:00:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001204467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.713623623s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 10:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.71365264s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................................XXXX] duration: 5.000655834s trace: https://jaeger.dev.storj.io/trace/727ebb7e32daec9f FAIL (2025-11-30 09:59:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000688844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.102370451s trace: https://jaeger.dev.storj.io/trace/2cc7b50bf9179eb4 FAIL (2025-11-30 09:59:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== AX3H7VAMFR3VM7NNJSRIXUD5PSRNJ4GK6MKHXCPUGI5OQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.030158449 (456h14m28.388817093s) [noise=true , address=77.90.31.21:4888 , piece_id=GNQRLBMRNTBA3K6JXZV4QOHR4YTZVFWTEEEBXS6UCPGLV3CZR6UA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.031256356 (456h14m28.389915019s) [noise=true , address=145.79.227.52:4459 , piece_id=RAV3SB3VDVBV6EDTJ2MCC2WHP4HVL4PEG5BAXMORA4YL36TVTLMQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.032116979 (456h14m28.390775623s) [noise=true , address=45.91.116.10:4085 , piece_id=UOVZSENGVMVYIWBSCSHJ543AXT7D7JCADMZ5DN33QYJIG2O2W5NQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.032241990 (456h14m28.390900634s) [noise=true , address=145.79.227.5:4234 , piece_id=VKRJRI5CHPDRFSZZMRE5B3Q74SHGD3FMSRHY3PYJUFHZSNQREFNQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.032727402 (456h14m28.391386059s) [noise=true , address=45.91.116.7:4037 , piece_id=P4LG3HULAGU5JDX7AQ4O2TT5IVU5BD4VY6KHFWIN7N77OIHG7ELQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.033264253 (456h14m28.391922961s) [noise=true , address=145.79.227.18:4015 , piece_id=4UHZW3G7ZRUH3GI5STCEP2Q3OUBIFPV5D7LT736R65RST774WWYA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.033414841 (456h14m28.392073497s) [noise=true , address=45.91.116.8:4068 , piece_id=2GYF2XAHE6PVABBKVOU6XNDJP5QOHKGMG4KGI6SVK5GFFHCQEFSQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.033873399 (456h14m28.392532065s) [noise=true , address=45.91.116.13:4176 , piece_id=GLKK2533FWO274V6YO6NDA7YGMT7C3JBBGTWDJMRKCFGVT63OGHQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.033989930 (456h14m28.392648593s) [noise=true , address=77.90.31.16:4757 , piece_id=K2FZMLWHFAOCCUECVCQXGEYMNL5VCDOEG6ASDAUBQEM7STXZ5POA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.034334921 (456h14m28.392993579s) [noise=true , address=45.91.116.12:4235 , piece_id=TCRLDLWCSBIPCAVV3ZPEJJKIZFVP7GPDEVWRC46RKBYDMVUY4H2Q, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.034380642 (456h14m28.393039299s) [noise=true , address=77.90.31.11:4580 , piece_id=UBNERJ4LZSLPS6VSVLDQ5AQTJ547PB2TLSIW2GDD3M6NMBMURMWA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.034761520 (456h14m28.393420178s) [noise=true , address=64.49.13.169:4711 , piece_id=C6N4NX22FUYTOWTG6A2GVF443FBV3LGNA3FE3FQNVEKUEXPDMX7A, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.034823070 (456h14m28.393481727s) [noise=true , address=45.91.116.6:4987 , piece_id=AU5YNDLOM5EARXJKIEV5XB3LQNLLXYHZMCH6CSIG6PXJRZ5KK2AA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.035147345 (456h14m28.393806013s) [noise=true , address=77.90.31.10:4531 , piece_id=FTTX7HKZ6MQASM3AYCBTWXUANWJDHD432ET3PKYL44BMM5HSPVWQ, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.035288520 (456h14m28.393947184s) [noise=true , address=145.79.227.12:4791 , piece_id=DIYQGILID73QSXFL3AMUCCN3Z2XVH4RN7ESJ43ZGBCQBRULU4DDA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.036323666 (456h14m28.394982337s) [noise=true , address=77.90.31.14:4692 , piece_id=67B3VGIR6ATBCYNJYKO2QD7RIC6SJRZ5W3Z33DJYAQNFOXHEQEIA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.036765497 (456h14m28.395424155s) [noise=true , address=77.90.31.12:4614 , piece_id=UN5E3KNJWX6T34MWOZA6JTVUQ4QGBENO3CEEJTLUZKMMGIU4YZUA, seg_pos={0 0}, upload=401840]: Uploading piece... Nov 30 09:58:43.595958651 (456h14m28.954617308s) [upload=401840]: Next returned nil segment Nov 30 09:58:43.595976096 (456h14m28.95463476s) [upload=401840]: Waiting for error group managing segments... Nov 30 09:58:43.735241460 (456h14m29.093900121s) [noise=true , address=145.79.227.52:4459 , piece_id=RAV3SB3VDVBV6EDTJ2MCC2WHP4HVL4PEG5BAXMORA4YL36TVTLMQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.735694980 (456h14m29.094353642s) [noise=true , address=145.79.227.18:4015 , piece_id=4UHZW3G7ZRUH3GI5STCEP2Q3OUBIFPV5D7LT736R65RST774WWYA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.738256403 (456h14m29.096915072s) [noise=true , address=45.91.116.10:4085 , piece_id=UOVZSENGVMVYIWBSCSHJ543AXT7D7JCADMZ5DN33QYJIG2O2W5NQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.742360435 (456h14m29.101019095s) [noise=true , address=45.91.116.6:4987 , piece_id=AU5YNDLOM5EARXJKIEV5XB3LQNLLXYHZMCH6CSIG6PXJRZ5KK2AA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.743369899 (456h14m29.102028562s) [noise=true , address=45.91.116.13:4176 , piece_id=GLKK2533FWO274V6YO6NDA7YGMT7C3JBBGTWDJMRKCFGVT63OGHQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.744602758 (456h14m29.103261422s) [noise=true , address=145.79.227.5:4234 , piece_id=VKRJRI5CHPDRFSZZMRE5B3Q74SHGD3FMSRHY3PYJUFHZSNQREFNQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.751075109 (456h14m29.109733769s) [noise=true , address=45.91.116.8:4068 , piece_id=2GYF2XAHE6PVABBKVOU6XNDJP5QOHKGMG4KGI6SVK5GFFHCQEFSQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.756292127 (456h14m29.114950799s) [noise=true , address=45.91.116.12:4235 , piece_id=TCRLDLWCSBIPCAVV3ZPEJJKIZFVP7GPDEVWRC46RKBYDMVUY4H2Q, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:43.763646708 (456h14m29.12230538s) [noise=true , address=45.91.116.7:4037 , piece_id=P4LG3HULAGU5JDX7AQ4O2TT5IVU5BD4VY6KHFWIN7N77OIHG7ELQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:45.053392026 (456h14m30.412050687s) [noise=true , address=145.79.227.12:4791 , piece_id=DIYQGILID73QSXFL3AMUCCN3Z2XVH4RN7ESJ43ZGBCQBRULU4DDA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:58:45.562527851 (456h14m30.92118651s) [noise=true , address=64.49.13.169:4711 , piece_id=C6N4NX22FUYTOWTG6A2GVF443FBV3LGNA3FE3FQNVEKUEXPDMX7A, seg_pos={0 0}, upload=401840]: Done uploading piece. err: <nil> Nov 30 09:59:42.652284597 (456h15m28.010943256s) [noise=true , address=77.90.31.17:4775 , piece_id=SO5XRTXAX3H7VAMFR3VM7NNJSRIXUD5PSRNJ4GK6MKHXCPUGI5OQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:127VKvwi82kYfGZ6Wcv8BinaNUQBMQsgKLKNNLht5JPsBLN7ovt, address:77.90.31.17:4775): protocol: write tcp 139.59.144.22:41162->77.90.31.17:4775: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652496928 (456h15m28.011155587s) [noise=true , address=77.90.31.10:4531 , piece_id=FTTX7HKZ6MQASM3AYCBTWXUANWJDHD432ET3PKYL44BMM5HSPVWQ, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: write tcp 139.59.144.22:39184->77.90.31.10:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652648913 (456h15m28.01130757s) [noise=true , address=77.90.31.14:4692 , piece_id=67B3VGIR6ATBCYNJYKO2QD7RIC6SJRZ5W3Z33DJYAQNFOXHEQEIA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T, address:77.90.31.14:4692): protocol: write tcp 139.59.144.22:60524->77.90.31.14:4692: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652721061 (456h15m28.011379717s) [noise=true , address=77.90.31.12:4614 , piece_id=UN5E3KNJWX6T34MWOZA6JTVUQ4QGBENO3CEEJTLUZKMMGIU4YZUA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 139.59.144.22:47762->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652836966 (456h15m28.011495625s) [noise=true , address=77.90.31.16:4757 , piece_id=K2FZMLWHFAOCCUECVCQXGEYMNL5VCDOEG6ASDAUBQEM7STXZ5POA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:41510->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652903342 (456h15m28.011561995s) [noise=true , address=77.90.31.21:4888 , piece_id=GNQRLBMRNTBA3K6JXZV4QOHR4YTZVFWTEEEBXS6UCPGLV3CZR6UA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: write tcp 139.59.144.22:34228->77.90.31.21:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652952286 (456h15m28.011610944s) [noise=true , address=77.90.31.11:4580 , piece_id=UBNERJ4LZSLPS6VSVLDQ5AQTJ547PB2TLSIW2GDD3M6NMBMURMWA, seg_pos={0 0}, upload=401840]: Done uploading piece. err: ecclient: upload failed (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, address:77.90.31.11:4580): protocol: write tcp 139.59.144.22:53158->77.90.31.11:4580: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:42.652989341 (456h15m28.011648003s) [seg_pos={0 0}, upload=401840]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 09:59:42.752947273 (456h15m28.111605927s) [upload=401840]: Done uploading segments... Nov 30 09:59:42.752979675 (456h15m28.111638332s) [upload=401840]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.102382719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.789780872s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 09:59:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789806031s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000892553s trace: https://jaeger.dev.storj.io/trace/486d2f59bfbfa726 FAIL (2025-11-30 09:59:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000903909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.568573847s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 09:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.568596046s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.112798891s trace: https://jaeger.dev.storj.io/trace/2749db3314a4cca0 FAIL (2025-11-30 09:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ad=401828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 30 09:57:42.689212185 (456h13m28.047870848s) [upload=401828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 09:57:42.800958613 (456h13m28.159617266s) [seg_pos={0 0}, upload=401828]: Begin upload segment... Nov 30 09:57:42.801019068 (456h13m28.159677715s) [seg_pos={0 0}, upload=401828]: Done begin upload segment. Nov 30 09:57:42.801056957 (456h13m28.159715613s) [noise=true , address=45.91.116.6:4996 , piece_id=VPZQ6DJ3PANOEJVOLSBE7T3XB3KWHUUQ5MXSJS77MYATG4HTQGAA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.801066699 (456h13m28.159725345s) [noise=true , address=77.90.31.19:4846 , piece_id=7FWJUNV2J2BDQHWMR6IAA5PM6GXXJXNGISSE4WFFGD3AVPMLH47Q, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.802010724 (456h13m28.160669383s) [noise=true , address=145.79.227.10:4422 , piece_id=DA2A64IWLEDL7A7HS7LCPWJBRPEETR5R6A34AKK7VEU6ALTPNCFA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.802418709 (456h13m28.161077371s) [noise=true , address=77.90.31.11:4579 , piece_id=NKCZ7NLDFIC24ROJNS2F5UCPAR744WF36FCJIQEVJFC5H4VW2SCQ, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.802506847 (456h13m28.161165504s) [noise=true , address=145.79.227.39:4046 , piece_id=F7GV7HSTNQBJEDNZHHSZIJZOBHLIL4BUZCIYSBEUURJFZO4VUPDA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.802817069 (456h13m28.161475729s) [noise=true , address=64.49.13.169:4708 , piece_id=N5IZOJ7WFSL3TSALP65QLPBFDKY4EL5ZHZF77LNMWC54G7SQT4ZA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.802852615 (456h13m28.161511268s) [noise=true , address=45.91.116.5:4320 , piece_id=H7SXZYXI5SZWFMFW4MQ6PXSVBEPHUF3OR5UO3QLEP62O6FVDSQJA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803137188 (456h13m28.161795842s) [noise=true , address=64.49.13.27:4968 , piece_id=BR2JR3JBGFLIPNBIUA2EXXYHELNCO643JJ76G57VJWEOS3ZLJXOQ, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803218634 (456h13m28.161877294s) [noise=true , address=77.90.31.18:4818 , piece_id=PES55V3DHMJ5R3OCZLJTGKK7BQQBHHHAW5W6VLKJ6V7G7X52O6TQ, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803412752 (456h13m28.162071408s) [noise=true , address=77.90.31.12:4608 , piece_id=5TOYCHZHEVPFTAQKJTIK4EACBTYMF2J243R2B27ZP4ISWU6UYD7Q, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803556082 (456h13m28.162214741s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZIYYTSIDTFWNA67XNLFS36YWBVSHSUKGXBLIS6E4OFVGXASPQSA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803672386 (456h13m28.162331042s) [noise=true , address=77.90.31.10:4532 , piece_id=6TA4V5IDNS7W5LA26XEGQO2ALOCRR4QFAM3WDIFS5ZXC4HEVIU4A, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.803937782 (456h13m28.162596435s) [noise=true , address=64.49.13.139:4478 , piece_id=RTSJUFFX7NMA6TMRFZ475GL5GGGRBF2YNTKHDYWFLSWFTFQPBDFA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.804247550 (456h13m28.162906209s) [noise=true , address=45.91.116.10:4109 , piece_id=VMNZ5DZAJJCYRXDIN2XBGEZH42XVR5KWWBW7KAZ6BDD3YKMUFEWA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.804631174 (456h13m28.163289834s) [noise=true , address=45.91.116.13:4186 , piece_id=KTIRKX53EKAQBBJGBH72RS6OZYCCG6CG2DWKLPLDV6SDNQM6FHVQ, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.804818242 (456h13m28.163476898s) [noise=true , address=77.90.31.21:4892 , piece_id=VSI37AFLLIBXNBKTCCBIETJOZAKCOKTMKBPBTGNJU46BMZ36YUNQ, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.805233003 (456h13m28.163891662s) [noise=true , address=45.91.116.8:4072 , piece_id=JOR6JYDG3YN7C25S2EP5SVTJJYUARLALQRK3M7KG4OQYU4T7LISA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:42.805676847 (456h13m28.164335507s) [noise=true , address=145.79.227.6:4285 , piece_id=IVQT7PHER2P43LP7SX7X3WYF6S6Y2V2XKXHNLPSBMZXBEI2VZ2BA, seg_pos={0 0}, upload=401828]: Uploading piece... Nov 30 09:57:43.352025878 (456h13m28.710684525s) [upload=401828]: Next returned nil segment Nov 30 09:57:43.352041237 (456h13m28.710699893s) [upload=401828]: Waiting for error group managing segments... Nov 30 09:57:43.497794773 (456h13m28.85645345s) [noise=true , address=45.91.116.5:4320 , piece_id=H7SXZYXI5SZWFMFW4MQ6PXSVBEPHUF3OR5UO3QLEP62O6FVDSQJA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.498844613 (456h13m28.857503261s) [noise=true , address=145.79.227.6:4285 , piece_id=IVQT7PHER2P43LP7SX7X3WYF6S6Y2V2XKXHNLPSBMZXBEI2VZ2BA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.500157352 (456h13m28.858816014s) [noise=true , address=145.79.227.10:4422 , piece_id=DA2A64IWLEDL7A7HS7LCPWJBRPEETR5R6A34AKK7VEU6ALTPNCFA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.501261442 (456h13m28.859920097s) [noise=true , address=45.91.116.6:4996 , piece_id=VPZQ6DJ3PANOEJVOLSBE7T3XB3KWHUUQ5MXSJS77MYATG4HTQGAA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.508524463 (456h13m28.867183118s) [noise=true , address=45.91.116.8:4072 , piece_id=JOR6JYDG3YN7C25S2EP5SVTJJYUARLALQRK3M7KG4OQYU4T7LISA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.511249474 (456h13m28.869908127s) [noise=true , address=145.79.227.39:4046 , piece_id=F7GV7HSTNQBJEDNZHHSZIJZOBHLIL4BUZCIYSBEUURJFZO4VUPDA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.513683845 (456h13m28.872342501s) [noise=true , address=45.91.116.13:4186 , piece_id=KTIRKX53EKAQBBJGBH72RS6OZYCCG6CG2DWKLPLDV6SDNQM6FHVQ, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:43.514578231 (456h13m28.873236885s) [noise=true , address=45.91.116.10:4109 , piece_id=VMNZ5DZAJJCYRXDIN2XBGEZH42XVR5KWWBW7KAZ6BDD3YKMUFEWA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:44.063878421 (456h13m29.42253709s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZIYYTSIDTFWNA67XNLFS36YWBVSHSUKGXBLIS6E4OFVGXASPQSA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:44.118822489 (456h13m29.477481151s) [noise=true , address=64.49.13.27:4968 , piece_id=BR2JR3JBGFLIPNBIUA2EXXYHELNCO643JJ76G57VJWEOS3ZLJXOQ, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:44.199600074 (456h13m29.558258729s) [noise=true , address=64.49.13.139:4478 , piece_id=RTSJUFFX7NMA6TMRFZ475GL5GGGRBF2YNTKHDYWFLSWFTFQPBDFA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:57:45.293144775 (456h13m30.651803436s) [noise=true , address=64.49.13.169:4708 , piece_id=N5IZOJ7WFSL3TSALP65QLPBFDKY4EL5ZHZF77LNMWC54G7SQT4ZA, seg_pos={0 0}, upload=401828]: Done uploading piece. err: <nil> Nov 30 09:58:42.399849942 (456h14m27.758508605s) [noise=true , address=77.90.31.18:4818 , piece_id=PES55V3DHMJ5R3OCZLJTGKK7BQQBHHHAW5W6VLKJ6V7G7X52O6TQ, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: rpc: dial tcp 77.90.31.18:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:42.400180382 (456h14m27.758839045s) [noise=true , address=77.90.31.21:4892 , piece_id=VSI37AFLLIBXNBKTCCBIETJOZAKCOKTMKBPBTGNJU46BMZ36YUNQ, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, address:77.90.31.21:4892): protocol: rpc: dial tcp 77.90.31.21:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:42.400415919 (456h14m27.759074583s) [noise=true , address=77.90.31.12:4608 , piece_id=5TOYCHZHEVPFTAQKJTIK4EACBTYMF2J243R2B27ZP4ISWU6UYD7Q, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:42.400633429 (456h14m27.759292092s) [noise=true , address=77.90.31.11:4579 , piece_id=NKCZ7NLDFIC24ROJNS2F5UCPAR744WF36FCJIQEVJFC5H4VW2SCQ, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: rpc: dial tcp 77.90.31.11:4579: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:42.400783687 (456h14m27.75944235s) [noise=true , address=77.90.31.19:4846 , piece_id=7FWJUNV2J2BDQHWMR6IAA5PM6GXXJXNGISSE4WFFGD3AVPMLH47Q, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 139.59.144.22:59844->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:42.401277623 (456h14m27.759936285s) [noise=true , address=77.90.31.10:4532 , piece_id=6TA4V5IDNS7W5LA26XEGQO2ALOCRR4QFAM3WDIFS5ZXC4HEVIU4A, seg_pos={0 0}, upload=401828]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: context deadline exceeded; context deadline exceeded Nov 30 09:58:42.401309087 (456h14m27.759967727s) [seg_pos={0 0}, upload=401828]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 09:58:42.510004738 (456h14m27.868663395s) [upload=401828]: Done uploading segments... Nov 30 09:58:42.510049713 (456h14m27.868708367s) [upload=401828]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.112820651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................................XXX] duration: 5.001467907s trace: https://jaeger.dev.storj.io/trace/d4430ee2928961d FAIL (2025-11-30 09:58:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001489428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000867218s trace: https://jaeger.dev.storj.io/trace/6f12b234ebacdf88 FAIL (2025-11-30 09:58:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000890462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001298772s trace: https://jaeger.dev.storj.io/trace/3d03fc52c5649e0b FAIL (2025-11-30 09:57:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001316216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.836618198s trace: https://jaeger.dev.storj.io/trace/14793eec43085e21 FAIL (2025-11-30 09:57:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.836715734s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001428689s trace: https://jaeger.dev.storj.io/trace/5c5cc3f4d3a26f17 FAIL (2025-11-30 09:57:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001842211s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)