Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.644807621s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:15:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.644834537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.69906141s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:15:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.69909135s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.693387231s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:15:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.693439323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 2.389679532s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.389711893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.742441822s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:15:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742485017s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.633780075s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:14:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.633805424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 2.369073406s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:14:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.369166012s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.669575711s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:14:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.669654811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 2.275549807s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.27557951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 2.180162655s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:13:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.180263071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.842279916s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:13:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.842319716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 2.365362508s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:13:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.365389626s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.653120595s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.653156326s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.62311112s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.623145704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.77005593s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:12:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.770081741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 2.399129754s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:12:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.399155459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.682830869s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:12:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.682855802s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.658244712s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:12:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658349522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.661644103s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.661671545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 2.36818661s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:11:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.368212689s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.597110996s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.597143247s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 2.191286674s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:11:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.1913142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.8196808s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:10:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819708645s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.740065718s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:10:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.740097523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 2.473557548s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:10:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.473648016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 2.532456031s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.532486633s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.593233703s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:09:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.5932711s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.684438676s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:09:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.684470594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 2.788505855s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.788538603s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.056275389s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:09:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.056306653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 2.274118325s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.274140419s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 2.255847122s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:08:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.255934636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.84376212s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:08:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.843789977s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.605702113s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.60572838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.773543954s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:07:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.773573414s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 2.525288105s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.525387574s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.636121915s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:07:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.636144377s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration: 2.606104817s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:07:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.606133763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.738422706s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:06:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.738682061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....X....................] duration: 2.330947718s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:06:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.330980181s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.618751753s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.618924844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.836747926s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.836805685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....X...................] duration: 2.441167433s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:05:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.441200813s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.704056546s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:05:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.704087101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 2.420438812s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:05:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.420540464s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.642497817s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:05:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.642531531s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 1.717447247s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:05:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.717582378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration: 2.37515048s trace: https://jaeger.dev.storj.io/trace/6daa2481c6dd1315 FAIL (2025-11-05 19:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 5 18:51:43.432244377 (151h10m46.133836829s) [upload=132977]: Starting upload Nov 5 18:51:43.432330480 (151h10m46.133922936s) [upload=132977]: Done starting upload Nov 5 18:51:43.433820526 (151h10m46.135412964s) [upload=132977]: Uploading segments... Nov 5 18:51:43.434260380 (151h10m46.135852834s) [upload=132977]: Got next segment. Inline: false Nov 5 18:51:43.434337735 (151h10m46.135930185s) [upload=132977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:51:43.434344454 (151h10m46.135936903s) [upload=132977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:51:43.546420493 (151h10m46.248012941s) [seg_pos={0 0}, upload=132977]: Begin upload segment... Nov 5 18:51:43.546508634 (151h10m46.248101082s) [seg_pos={0 0}, upload=132977]: Done begin upload segment. Nov 5 18:51:43.546549698 (151h10m46.248142152s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.546614359 (151h10m46.248206807s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547543878 (151h10m46.249136337s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.547586023 (151h10m46.249178482s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548063269 (151h10m46.249655739s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548091497 (151h10m46.249683949s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548514948 (151h10m46.250107402s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548622130 (151h10m46.250214587s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.548941852 (151h10m46.250534305s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549297422 (151h10m46.250889885s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549414927 (151h10m46.251007387s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549841517 (151h10m46.251433975s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.549899813 (151h10m46.251492271s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.551899136 (151h10m46.253491596s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.553177767 (151h10m46.254770225s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:43.554870320 (151h10m46.256462779s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Uploading piece... Nov 5 18:51:44.157123531 (151h10m46.858715966s) [upload=132977]: Next returned nil segment Nov 5 18:51:44.157136380 (151h10m46.858728814s) [upload=132977]: Waiting for error group managing segments... Nov 5 18:51:44.294257785 (151h10m46.995850241s) [noise=true , address=145.79.227.26:4653 , piece_id=BWUG7RZ6I2AJTW75E5QNZCEH3DT4KYWIMIWMO5B2BJFEL63AO6JA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296061559 (151h10m46.99765401s) [noise=true , address=145.79.227.24:4583 , piece_id=5IYHH3F4IDC4S4BHSYLFPFWGLRXNM4VZTWWTT2VGPBBYX3HXIMFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.296663568 (151h10m46.99825602s) [noise=true , address=145.79.227.13:4802 , piece_id=YWJXKTVSDXB3LRF3GAHJDEGN7QWHVNMJVMF3C4FJYIAUCQX6RJ5Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.299751559 (151h10m47.001344021s) [noise=true , address=145.79.227.18:4005 , piece_id=2UBFYPDX4CNKLGWMCGDULFBIFAKZJ7R46NPB6ETBINLGXG5SELBA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.301207752 (151h10m47.002800207s) [noise=true , address=145.79.227.28:4726 , piece_id=RVQUAYSISAYKAOERHEA3GHT4JDKCYQ4VIUJAPTCLPA2AUUMYF3EA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307278288 (151h10m47.008870728s) [noise=true , address=145.79.227.9:4367 , piece_id=KMVPASKTBBW2HVLHKDOTUN425ICHP3EC7EOZR5WWK2SMMWVHVE4Q, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.307882505 (151h10m47.009474953s) [noise=true , address=145.79.227.10:4402 , piece_id=TH43VATKUZOFI7FER5NR4IVJCQXW42FVVBC74XABOP6PATTWW4RA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.308501134 (151h10m47.010093583s) [noise=true , address=145.79.227.31:4799 , piece_id=LMVE4FZFWWLPNTLXNSCGZBJ4HWEA3RRZBV73EEUIGTBQCCR46UXA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311414279 (151h10m47.013006732s) [noise=true , address=145.79.227.20:4439 , piece_id=Y4THAMVSWIMSCOAT2THQBI5C4RMGYUVTQTSOHDNKZSMJNQJR2DGA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.311558970 (151h10m47.013151423s) [noise=true , address=145.79.227.27:4691 , piece_id=6L4SIH4CCUQGXW2I5CIJTLMDT65IEYUPFFKBZOCHNDNNPMDIV6WA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:51:44.312109406 (151h10m47.013701857s) [noise=true , address=145.79.227.8:4329 , piece_id=L7YOW45AIYL5QJWNRTKMPECQ7L7YVXEX3JVZF5VXOEM5EFUO6X3A, seg_pos={0 0}, upload=132977]: Done uploading piece. err: <nil> Nov 5 18:52:13.143676122 (151h11m15.845268581s) [noise=true , address=77.90.31.13:4627 , piece_id=GFNJH2RE4U2DWKBIEBRSS2RZPAGXEZXVGQ62AIR44JMTWDVOLMEA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53200->77.90.31.13:4627: use of closed network connection Nov 5 18:52:13.143751291 (151h11m15.845343747s) [noise=true , address=77.90.31.11:4554 , piece_id=ORMUMOFDWMTEUOPUKJEWKTLJ3X7U4THRQTTA6DUSZOXFO2IIBQNA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40474->77.90.31.11:4554: use of closed network connection Nov 5 18:52:13.143980660 (151h11m15.845573114s) [noise=true , address=45.91.116.8:4047 , piece_id=XF3KR4AWS2DU6XATELGS3RNUZ2HHJM2ZYLSF67QPYX6KSIQBXNZA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:53756->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144324074 (151h11m15.845916522s) [noise=true , address=45.91.116.6:4974 , piece_id=O4IATMIYTLAAC7PQJO2AZIUVIETIYDWAH73GMEIMVIDUAB2LR7CA, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 139.59.144.22:43348->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144333953 (151h11m15.845926412s) [noise=true , address=77.90.31.21:4879 , piece_id=ULWU2WFDHVEN4V3YRDRWNP2ZQ6ECF2TLYSPCIAYEY7BZQLKXPIFQ, seg_pos={0 0}, upload=132977]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:44726->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:52:13.144389532 (151h11m15.845981968s) [seg_pos={0 0}, upload=132977]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 5 18:52:13.252658837 (151h11m15.954251274s) [upload=132977]: Done uploading segments... Nov 5 18:52:13.252693157 (151h11m15.954285608s) [upload=132977]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.375187034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.669897327s trace: https://jaeger.dev.storj.io/trace/72a639688f7f6d0b FAIL (2025-11-05 19:04:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.66993758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.909993482s trace: https://jaeger.dev.storj.io/trace/449848d9d77db041 FAIL (2025-11-05 19:04:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.910200102s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)