Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 2.924201924s FAIL (2025-08-19 16:54:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.924226417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 2.94166073s FAIL (2025-08-19 16:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.941713413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X............X......................................X...] duration: 1.773984601s FAIL (2025-08-19 16:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 16:51:17.946611843 (263h25m39.905709076s) [upload=274249]: Starting upload Aug 19 16:51:17.946668750 (263h25m39.905766001s) [upload=274249]: Done starting upload Aug 19 16:51:17.946743651 (263h25m39.905840882s) [upload=274249]: Uploading segments... Aug 19 16:51:17.946797160 (263h25m39.905894408s) [upload=274249]: Got next segment. Inline: true Aug 19 16:51:17.946802254 (263h25m39.905899499s) [upload=274249]: Waiting for error group managing segments... Aug 19 16:51:17.946838751 (263h25m39.905935996s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:51:17.946842821 (263h25m39.905940066s) [upload=274249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:51:17.946845634 (263h25m39.90594288s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:51:18.039518081 (263h25m39.998615398s) [upload=274249]: Done uploading segments... Aug 19 16:51:18.039565782 (263h25m39.998663047s) [upload=274249]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.774014911s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X....XXX............] duration: 2.496142812s FAIL (2025-08-19 16:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.496247592s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 2.712368489s FAIL (2025-08-19 16:53:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.712400219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X......................................X..] duration: 1.034031647s FAIL (2025-08-19 16:53:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 16:51:17.946611843 (263h25m39.905709076s) [upload=274249]: Starting upload Aug 19 16:51:17.946668750 (263h25m39.905766001s) [upload=274249]: Done starting upload Aug 19 16:51:17.946743651 (263h25m39.905840882s) [upload=274249]: Uploading segments... Aug 19 16:51:17.946797160 (263h25m39.905894408s) [upload=274249]: Got next segment. Inline: true Aug 19 16:51:17.946802254 (263h25m39.905899499s) [upload=274249]: Waiting for error group managing segments... Aug 19 16:51:17.946838751 (263h25m39.905935996s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:51:17.946842821 (263h25m39.905940066s) [upload=274249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:51:17.946845634 (263h25m39.90594288s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:51:18.039518081 (263h25m39.998615398s) [upload=274249]: Done uploading segments... Aug 19 16:51:18.039565782 (263h25m39.998663047s) [upload=274249]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.034065057s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 2.495187708s FAIL (2025-08-19 16:53:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.495229561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XX...XX....] duration: 2.945430668s FAIL (2025-08-19 16:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.945526198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 2.467456279s FAIL (2025-08-19 16:52:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.467483582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X....XXX...........] duration: 3.347418017s FAIL (2025-08-19 16:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 3.347446342s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X............X......................................X.] duration: 829.120758ms FAIL (2025-08-19 16:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 16:51:17.946611843 (263h25m39.905709076s) [upload=274249]: Starting upload Aug 19 16:51:17.946668750 (263h25m39.905766001s) [upload=274249]: Done starting upload Aug 19 16:51:17.946743651 (263h25m39.905840882s) [upload=274249]: Uploading segments... Aug 19 16:51:17.946797160 (263h25m39.905894408s) [upload=274249]: Got next segment. Inline: true Aug 19 16:51:17.946802254 (263h25m39.905899499s) [upload=274249]: Waiting for error group managing segments... Aug 19 16:51:17.946838751 (263h25m39.905935996s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:51:17.946842821 (263h25m39.905940066s) [upload=274249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:51:17.946845634 (263h25m39.90594288s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:51:18.039518081 (263h25m39.998615398s) [upload=274249]: Done uploading segments... Aug 19 16:51:18.039565782 (263h25m39.998663047s) [upload=274249]: Upload finished. err: metaclient: Too Many Requests Duration:* 829.153795ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X....X....] duration: 2.849845173s FAIL (2025-08-19 16:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.849878071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XX...XX...] duration: 2.836391018s FAIL (2025-08-19 16:52:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.836419691s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 2.459128196s FAIL (2025-08-19 16:51:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.459156825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X....XXX..........] duration: 3.158637973s FAIL (2025-08-19 16:51:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 3.158729627s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 2.602856624s FAIL (2025-08-19 16:51:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.602887892s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX...XX..] duration: 2.683558053s FAIL (2025-08-19 16:51:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.683594231s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X......................................X] duration: 383.1531ms FAIL (2025-08-19 16:51:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 16:51:17.946611843 (263h25m39.905709076s) [upload=274249]: Starting upload Aug 19 16:51:17.946668750 (263h25m39.905766001s) [upload=274249]: Done starting upload Aug 19 16:51:17.946743651 (263h25m39.905840882s) [upload=274249]: Uploading segments... Aug 19 16:51:17.946797160 (263h25m39.905894408s) [upload=274249]: Got next segment. Inline: true Aug 19 16:51:17.946802254 (263h25m39.905899499s) [upload=274249]: Waiting for error group managing segments... Aug 19 16:51:17.946838751 (263h25m39.905935996s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:51:17.946842821 (263h25m39.905940066s) [upload=274249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:51:17.946845634 (263h25m39.90594288s) [upload=274249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:51:18.039518081 (263h25m39.998615398s) [upload=274249]: Done uploading segments... Aug 19 16:51:18.039565782 (263h25m39.998663047s) [upload=274249]: Upload finished. err: metaclient: Too Many Requests Duration:* 383.179653ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 2.950824464s FAIL (2025-08-19 16:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.950865773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X....XXX.........] duration: 2.517332273s FAIL (2025-08-19 16:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.517365041s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 3.127588438s FAIL (2025-08-19 16:51:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.127623937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X....XXX........] duration: 2.697815112s FAIL (2025-08-19 16:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.697844247s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 2.382527672s FAIL (2025-08-19 16:50:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.38256293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X............X......................................] duration: 1.469899338s FAIL (2025-08-19 16:50:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 1.469928139s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 3.220443828s FAIL (2025-08-19 16:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.220477178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX...XX.] duration: 2.974572173s FAIL (2025-08-19 16:50:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.974599255s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 13.810856432s FAIL (2025-08-19 16:50:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 13.810896627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 7.677929583s FAIL (2025-08-19 16:49:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.677955966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X....XXX.......] duration: 8.311869253s FAIL (2025-08-19 16:49:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 8.311904344s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 1m0.120096917s FAIL (2025-08-19 16:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.120130274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XX...XX] duration: 30.001344333s FAIL (2025-08-19 16:49:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001370829s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X............X.....................................] duration: 965.779543ms FAIL (2025-08-19 16:49:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 965.860948ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.105493809s FAIL (2025-08-19 16:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.105520994s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XX...X] duration: 30.000647649s FAIL (2025-08-19 16:49:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000747066s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X....XXX......] duration: 2.999572887s FAIL (2025-08-19 16:48:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.999617547s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X............X....................................] duration: 1.229732193s FAIL (2025-08-19 16:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 1.229771055s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X............X...................................] duration: 855.911796ms FAIL (2025-08-19 16:47:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 855.938191ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.507398154s FAIL (2025-08-19 16:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382251427 (263h18m59.341348675s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382414984 (263h18m59.341512232s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382673030 (263h18m59.341770276s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382735719 (263h18m59.341832965s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383053107 (263h18m59.34215045s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383084717 (263h18m59.342181964s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383534576 (263h18m59.342631858s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383917134 (263h18m59.343014388s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384348854 (263h18m59.343446109s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384718383 (263h18m59.343815637s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385071761 (263h18m59.344169015s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385434046 (263h18m59.344531298s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385721240 (263h18m59.344818486s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.386090919 (263h18m59.345188282s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.964767860 (263h18m59.923865117s) [upload=274134]: Next returned nil segment Aug 19 16:44:37.964798911 (263h18m59.923896169s) [upload=274134]: Waiting for error group managing segments... Aug 19 16:44:38.110619970 (263h19m0.069717223s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.147656211 (263h19m0.106753456s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.151775591 (263h19m0.110872839s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.154994774 (263h19m0.114092024s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.157317429 (263h19m0.116414677s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.162078807 (263h19m0.121176058s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.184864439 (263h19m0.143961692s) [noise=true , address=104.36.82.39:4738 , piece_id=FBW64WJA6ARTNDKWRXCTJC7PK2WP5KRBUZ3EJVP3HVWTB5G7B5IA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.296041507 (263h19m0.255138762s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:39.007977043 (263h19m0.967074295s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:45:36.900803755 (263h19m58.859901013s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: read tcp 139.59.144.22:47708->77.90.31.15:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.900894139 (263h19m58.859991396s) [noise=true , address=45.91.116.8:4068 , piece_id=2CZMYUU6FSTFLXFMH7ZAW3XE4N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82, address:45.91.116.8:4068): protocol: read tcp 139.59.144.22:51570->45.91.116.8:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901206820 (263h19m58.860304076s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: write tcp 139.59.144.22:54138->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901449505 (263h19m58.860546763s) [noise=true , address=77.90.31.21:4886 , piece_id=6HGSVG3VGLNSZSFYKNR4KEUXWS366XVYRE5HCP45RGJTYCVZMZ5A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection Aug 19 16:45:36.901524916 (263h19m58.860622165s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: write tcp 139.59.144.22:34756->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901587712 (263h19m58.860684948s) [noise=true , address=45.91.116.11:4123 , piece_id=RK4R3IGJWRZMVCMKEBD3KCB4NPIM53MX34PCEL3MQSZRCS7I6T2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12kjHfRVsDASQz36zUAHrXsLJausE7UTNzHgiFWpys4bQK21cAE, address:45.91.116.11:4123): protocol: read tcp 139.59.144.22:52338->45.91.116.11:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901649530 (263h19m58.860746782s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:47534->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901713013 (263h19m58.860810272s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: read tcp 139.59.144.22:35448->77.90.31.11:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901775369 (263h19m58.860872604s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: read tcp 139.59.144.22:33368->77.90.31.10:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901796067 (263h19m58.860893328s) [seg_pos={0 0}, upload=274134]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 19 16:45:37.011173029 (263h19m58.970270282s) [upload=274134]: Done uploading segments... Aug 19 16:45:37.011204568 (263h19m58.970301824s) [upload=274134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.507430951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X....XXX.....] duration: 2.95947898s FAIL (2025-08-19 16:47:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.959509812s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.531336164s FAIL (2025-08-19 16:47:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== [upload=274142]: Uploading segments... Aug 19 16:45:04.877751097 (263h19m26.836848377s) [upload=274142]: Got next segment. Inline: false Aug 19 16:45:04.877836551 (263h19m26.836933808s) [upload=274142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:45:04.877844126 (263h19m26.836941389s) [upload=274142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:45:04.986216040 (263h19m26.945313306s) [seg_pos={0 0}, upload=274142]: Begin upload segment... Aug 19 16:45:04.986305209 (263h19m26.945402469s) [seg_pos={0 0}, upload=274142]: Done begin upload segment. Aug 19 16:45:04.986364206 (263h19m26.945461464s) [noise=true , address=77.90.31.16:4735 , piece_id=4BINUM36RCVMJNSGOS2K2Y4V4NY3FYFTGE2LLLAMRZTI74YLAJ2A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.986905074 (263h19m26.946002335s) [noise=true , address=104.36.82.2:4730 , piece_id=ODA7JL4SGFO23SLP3BTCRTAKQOTZMC5M4IY5DF5IFZEMUBH7A7XQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.987361736 (263h19m26.946458994s) [noise=true , address=104.36.82.12:4151 , piece_id=ZXMRB2OWFIALMJIY7M5TVGH6QB435R3C4KBQAXV5K4REYQ7D4YCA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.987791403 (263h19m26.946888663s) [noise=true , address=104.36.82.11:4115 , piece_id=3I4Q3QL7XMIVJP2MEGDSVKUANIDWU7HIJV4HK5QW2K7PSUBH674A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988171984 (263h19m26.947269245s) [noise=true , address=77.90.31.19:4841 , piece_id=V4IQZZIOEUNRM2UZQM43LVAU7DD2Z6GCGXDV2ELDOA4SX7SRUYSQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988579037 (263h19m26.947676296s) [noise=true , address=45.91.116.7:4011 , piece_id=SCM2R7MZ7BXHKAC3HZJ2RO46AY256MAULWCCYWRROZK7GM2VSGHA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988960044 (263h19m26.948057305s) [noise=true , address=104.36.82.8:4970 , piece_id=UVCDEDYGRU2MRGARTHX6HE37MN7R7ENCKECID43OY5AKGRGUV4OA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.989308262 (263h19m26.948405523s) [noise=true , address=104.36.82.10:4078 , piece_id=YWBZ7RLCNXQ5GWNF3GXBBYKF67NZ3GMWERZNCOSZEOJVRD4NRDMQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990040369 (263h19m26.949137628s) [noise=true , address=45.91.116.13:4154 , piece_id=EWEONRKBR5FWUKRYS3XKZVBKNXDNAZ7N7YTNFIB7N6Z4NHMM7UZQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990461013 (263h19m26.949558276s) [noise=true , address=104.36.82.6:4874 , piece_id=QWVSGKI7TMJ4WANYEGXAEELGZ56MABCNHMMHB5BNRYQPMXWCZLPQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990837151 (263h19m26.949934427s) [noise=true , address=104.36.82.16:4293 , piece_id=U4XH5LDILLF4PYKLYM3VW57C4UWQOHKE6K6FK6QKGQV2GE4U4CNQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.991468207 (263h19m26.950565469s) [noise=true , address=104.36.82.14:4222 , piece_id=HP2X325S72BLJJJASFCRR5JM3CHKNGMDGLYWOJP4GFGTB6BWP7CA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.991860660 (263h19m26.950957937s) [noise=true , address=104.36.82.7:4910 , piece_id=UWUAEBX3AQLRMZ4PISHUI2MCGTYZKGQ4NKIIP2SMJJRESYVPBHBA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.992269680 (263h19m26.951366939s) [noise=true , address=77.90.31.18:4807 , piece_id=WQHNTCFW2YOSYYHNJ7HYTJ74PLJKIKXVTBUKGYMHN65LD7XFPC5Q, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.992702107 (263h19m26.951799366s) [noise=true , address=45.91.116.11:4118 , piece_id=O4JFCMWX3MAMQVTF7MTCY33WLOH6SK6ODNK4BP5KZQJ22OJOIBGQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.993146687 (263h19m26.952243945s) [noise=true , address=45.91.116.9:4263 , piece_id=WRYOPKJBZPYLLBEX5NS23237O57QTFOHO73HG4QH5SWBCUHHGN6A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:05.135668753 (263h19m27.094765995s) [noise=true , address=104.36.82.16:4293 , piece_id=U4XH5LDILLF4PYKLYM3VW57C4UWQOHKE6K6FK6QKGQV2GE4U4CNQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:39608->104.36.82.16:4293: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:05.689547414 (263h19m27.648644661s) [upload=274142]: Next returned nil segment Aug 19 16:45:05.689569931 (263h19m27.648667181s) [upload=274142]: Waiting for error group managing segments... Aug 19 16:45:05.821355785 (263h19m27.780453037s) [noise=true , address=104.36.82.12:4151 , piece_id=ZXMRB2OWFIALMJIY7M5TVGH6QB435R3C4KBQAXV5K4REYQ7D4YCA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.824077115 (263h19m27.783174363s) [noise=true , address=104.36.82.14:4222 , piece_id=HP2X325S72BLJJJASFCRR5JM3CHKNGMDGLYWOJP4GFGTB6BWP7CA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.826595205 (263h19m27.785692457s) [noise=true , address=104.36.82.10:4078 , piece_id=YWBZ7RLCNXQ5GWNF3GXBBYKF67NZ3GMWERZNCOSZEOJVRD4NRDMQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.832009226 (263h19m27.791106472s) [noise=true , address=104.36.82.7:4910 , piece_id=UWUAEBX3AQLRMZ4PISHUI2MCGTYZKGQ4NKIIP2SMJJRESYVPBHBA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.840463532 (263h19m27.799560781s) [noise=true , address=104.36.82.11:4115 , piece_id=3I4Q3QL7XMIVJP2MEGDSVKUANIDWU7HIJV4HK5QW2K7PSUBH674A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.856426909 (263h19m27.815524155s) [noise=true , address=104.36.82.2:4730 , piece_id=ODA7JL4SGFO23SLP3BTCRTAKQOTZMC5M4IY5DF5IFZEMUBH7A7XQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.860656125 (263h19m27.819753375s) [noise=true , address=104.36.82.8:4970 , piece_id=UVCDEDYGRU2MRGARTHX6HE37MN7R7ENCKECID43OY5AKGRGUV4OA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.892818451 (263h19m27.851915686s) [noise=true , address=104.36.82.6:4874 , piece_id=QWVSGKI7TMJ4WANYEGXAEELGZ56MABCNHMMHB5BNRYQPMXWCZLPQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:34.599483064 (263h19m56.55858031s) [noise=true , address=77.90.31.19:4841 , piece_id=V4IQZZIOEUNRM2UZQM43LVAU7DD2Z6GCGXDV2ELDOA4SX7SRUYSQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:45:34.599584471 (263h19m56.558681702s) [noise=true , address=77.90.31.16:4735 , piece_id=4BINUM36RCVMJNSGOS2K2Y4V4NY3FYFTGE2LLLAMRZTI74YLAJ2A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 139.59.144.22:44230->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599644420 (263h19m56.558741652s) [noise=true , address=45.91.116.13:4154 , piece_id=EWEONRKBR5FWUKRYS3XKZVBKNXDNAZ7N7YTNFIB7N6Z4NHMM7UZQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: read tcp 139.59.144.22:49616->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599689167 (263h19m56.558786398s) [noise=true , address=45.91.116.9:4263 , piece_id=WRYOPKJBZPYLLBEX5NS23237O57QTFOHO73HG4QH5SWBCUHHGN6A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33984->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599737297 (263h19m56.558834528s) [noise=true , address=45.91.116.7:4011 , piece_id=SCM2R7MZ7BXHKAC3HZJ2RO46AY256MAULWCCYWRROZK7GM2VSGHA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: read tcp 139.59.144.22:39790->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599743407 (263h19m56.558840639s) [noise=true , address=45.91.116.11:4118 , piece_id=O4JFCMWX3MAMQVTF7MTCY33WLOH6SK6ODNK4BP5KZQJ22OJOIBGQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 139.59.144.22:53524->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599799239 (263h19m56.558896488s) [noise=true , address=77.90.31.18:4807 , piece_id=WQHNTCFW2YOSYYHNJ7HYTJ74PLJKIKXVTBUKGYMHN65LD7XFPC5Q, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 139.59.144.22:54628->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599857068 (263h19m56.558954325s) [seg_pos={0 0}, upload=274142]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:34.724663931 (263h19m56.683761189s) [upload=274142]: Done uploading segments... Aug 19 16:45:34.724702399 (263h19m56.683799654s) [upload=274142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 3.531369914s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.776974566s FAIL (2025-08-19 16:47:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.776995484s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.514226111s FAIL (2025-08-19 16:47:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382251427 (263h18m59.341348675s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382414984 (263h18m59.341512232s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382673030 (263h18m59.341770276s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382735719 (263h18m59.341832965s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383053107 (263h18m59.34215045s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383084717 (263h18m59.342181964s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383534576 (263h18m59.342631858s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383917134 (263h18m59.343014388s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384348854 (263h18m59.343446109s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384718383 (263h18m59.343815637s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385071761 (263h18m59.344169015s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385434046 (263h18m59.344531298s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385721240 (263h18m59.344818486s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.386090919 (263h18m59.345188282s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.964767860 (263h18m59.923865117s) [upload=274134]: Next returned nil segment Aug 19 16:44:37.964798911 (263h18m59.923896169s) [upload=274134]: Waiting for error group managing segments... Aug 19 16:44:38.110619970 (263h19m0.069717223s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.147656211 (263h19m0.106753456s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.151775591 (263h19m0.110872839s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.154994774 (263h19m0.114092024s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.157317429 (263h19m0.116414677s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.162078807 (263h19m0.121176058s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.184864439 (263h19m0.143961692s) [noise=true , address=104.36.82.39:4738 , piece_id=FBW64WJA6ARTNDKWRXCTJC7PK2WP5KRBUZ3EJVP3HVWTB5G7B5IA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.296041507 (263h19m0.255138762s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:39.007977043 (263h19m0.967074295s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:45:36.900803755 (263h19m58.859901013s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: read tcp 139.59.144.22:47708->77.90.31.15:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.900894139 (263h19m58.859991396s) [noise=true , address=45.91.116.8:4068 , piece_id=2CZMYUU6FSTFLXFMH7ZAW3XE4N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82, address:45.91.116.8:4068): protocol: read tcp 139.59.144.22:51570->45.91.116.8:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901206820 (263h19m58.860304076s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: write tcp 139.59.144.22:54138->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901449505 (263h19m58.860546763s) [noise=true , address=77.90.31.21:4886 , piece_id=6HGSVG3VGLNSZSFYKNR4KEUXWS366XVYRE5HCP45RGJTYCVZMZ5A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection Aug 19 16:45:36.901524916 (263h19m58.860622165s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: write tcp 139.59.144.22:34756->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901587712 (263h19m58.860684948s) [noise=true , address=45.91.116.11:4123 , piece_id=RK4R3IGJWRZMVCMKEBD3KCB4NPIM53MX34PCEL3MQSZRCS7I6T2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12kjHfRVsDASQz36zUAHrXsLJausE7UTNzHgiFWpys4bQK21cAE, address:45.91.116.11:4123): protocol: read tcp 139.59.144.22:52338->45.91.116.11:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901649530 (263h19m58.860746782s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:47534->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901713013 (263h19m58.860810272s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: read tcp 139.59.144.22:35448->77.90.31.11:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901775369 (263h19m58.860872604s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: read tcp 139.59.144.22:33368->77.90.31.10:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901796067 (263h19m58.860893328s) [seg_pos={0 0}, upload=274134]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 19 16:45:37.011173029 (263h19m58.970270282s) [upload=274134]: Done uploading segments... Aug 19 16:45:37.011204568 (263h19m58.970301824s) [upload=274134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.514257635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X............X..................................] duration: 816.747808ms FAIL (2025-08-19 16:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 816.789178ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 2.657356707s FAIL (2025-08-19 16:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.657586187s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X....XXX....] duration: 2.54470172s FAIL (2025-08-19 16:46:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.544738622s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.55796169s FAIL (2025-08-19 16:46:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== [upload=274142]: Uploading segments... Aug 19 16:45:04.877751097 (263h19m26.836848377s) [upload=274142]: Got next segment. Inline: false Aug 19 16:45:04.877836551 (263h19m26.836933808s) [upload=274142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:45:04.877844126 (263h19m26.836941389s) [upload=274142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:45:04.986216040 (263h19m26.945313306s) [seg_pos={0 0}, upload=274142]: Begin upload segment... Aug 19 16:45:04.986305209 (263h19m26.945402469s) [seg_pos={0 0}, upload=274142]: Done begin upload segment. Aug 19 16:45:04.986364206 (263h19m26.945461464s) [noise=true , address=77.90.31.16:4735 , piece_id=4BINUM36RCVMJNSGOS2K2Y4V4NY3FYFTGE2LLLAMRZTI74YLAJ2A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.986905074 (263h19m26.946002335s) [noise=true , address=104.36.82.2:4730 , piece_id=ODA7JL4SGFO23SLP3BTCRTAKQOTZMC5M4IY5DF5IFZEMUBH7A7XQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.987361736 (263h19m26.946458994s) [noise=true , address=104.36.82.12:4151 , piece_id=ZXMRB2OWFIALMJIY7M5TVGH6QB435R3C4KBQAXV5K4REYQ7D4YCA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.987791403 (263h19m26.946888663s) [noise=true , address=104.36.82.11:4115 , piece_id=3I4Q3QL7XMIVJP2MEGDSVKUANIDWU7HIJV4HK5QW2K7PSUBH674A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988171984 (263h19m26.947269245s) [noise=true , address=77.90.31.19:4841 , piece_id=V4IQZZIOEUNRM2UZQM43LVAU7DD2Z6GCGXDV2ELDOA4SX7SRUYSQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988579037 (263h19m26.947676296s) [noise=true , address=45.91.116.7:4011 , piece_id=SCM2R7MZ7BXHKAC3HZJ2RO46AY256MAULWCCYWRROZK7GM2VSGHA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.988960044 (263h19m26.948057305s) [noise=true , address=104.36.82.8:4970 , piece_id=UVCDEDYGRU2MRGARTHX6HE37MN7R7ENCKECID43OY5AKGRGUV4OA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.989308262 (263h19m26.948405523s) [noise=true , address=104.36.82.10:4078 , piece_id=YWBZ7RLCNXQ5GWNF3GXBBYKF67NZ3GMWERZNCOSZEOJVRD4NRDMQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990040369 (263h19m26.949137628s) [noise=true , address=45.91.116.13:4154 , piece_id=EWEONRKBR5FWUKRYS3XKZVBKNXDNAZ7N7YTNFIB7N6Z4NHMM7UZQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990461013 (263h19m26.949558276s) [noise=true , address=104.36.82.6:4874 , piece_id=QWVSGKI7TMJ4WANYEGXAEELGZ56MABCNHMMHB5BNRYQPMXWCZLPQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.990837151 (263h19m26.949934427s) [noise=true , address=104.36.82.16:4293 , piece_id=U4XH5LDILLF4PYKLYM3VW57C4UWQOHKE6K6FK6QKGQV2GE4U4CNQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.991468207 (263h19m26.950565469s) [noise=true , address=104.36.82.14:4222 , piece_id=HP2X325S72BLJJJASFCRR5JM3CHKNGMDGLYWOJP4GFGTB6BWP7CA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.991860660 (263h19m26.950957937s) [noise=true , address=104.36.82.7:4910 , piece_id=UWUAEBX3AQLRMZ4PISHUI2MCGTYZKGQ4NKIIP2SMJJRESYVPBHBA, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.992269680 (263h19m26.951366939s) [noise=true , address=77.90.31.18:4807 , piece_id=WQHNTCFW2YOSYYHNJ7HYTJ74PLJKIKXVTBUKGYMHN65LD7XFPC5Q, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.992702107 (263h19m26.951799366s) [noise=true , address=45.91.116.11:4118 , piece_id=O4JFCMWX3MAMQVTF7MTCY33WLOH6SK6ODNK4BP5KZQJ22OJOIBGQ, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:04.993146687 (263h19m26.952243945s) [noise=true , address=45.91.116.9:4263 , piece_id=WRYOPKJBZPYLLBEX5NS23237O57QTFOHO73HG4QH5SWBCUHHGN6A, seg_pos={0 0}, upload=274142]: Uploading piece... Aug 19 16:45:05.135668753 (263h19m27.094765995s) [noise=true , address=104.36.82.16:4293 , piece_id=U4XH5LDILLF4PYKLYM3VW57C4UWQOHKE6K6FK6QKGQV2GE4U4CNQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:39608->104.36.82.16:4293: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:05.689547414 (263h19m27.648644661s) [upload=274142]: Next returned nil segment Aug 19 16:45:05.689569931 (263h19m27.648667181s) [upload=274142]: Waiting for error group managing segments... Aug 19 16:45:05.821355785 (263h19m27.780453037s) [noise=true , address=104.36.82.12:4151 , piece_id=ZXMRB2OWFIALMJIY7M5TVGH6QB435R3C4KBQAXV5K4REYQ7D4YCA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.824077115 (263h19m27.783174363s) [noise=true , address=104.36.82.14:4222 , piece_id=HP2X325S72BLJJJASFCRR5JM3CHKNGMDGLYWOJP4GFGTB6BWP7CA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.826595205 (263h19m27.785692457s) [noise=true , address=104.36.82.10:4078 , piece_id=YWBZ7RLCNXQ5GWNF3GXBBYKF67NZ3GMWERZNCOSZEOJVRD4NRDMQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.832009226 (263h19m27.791106472s) [noise=true , address=104.36.82.7:4910 , piece_id=UWUAEBX3AQLRMZ4PISHUI2MCGTYZKGQ4NKIIP2SMJJRESYVPBHBA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.840463532 (263h19m27.799560781s) [noise=true , address=104.36.82.11:4115 , piece_id=3I4Q3QL7XMIVJP2MEGDSVKUANIDWU7HIJV4HK5QW2K7PSUBH674A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.856426909 (263h19m27.815524155s) [noise=true , address=104.36.82.2:4730 , piece_id=ODA7JL4SGFO23SLP3BTCRTAKQOTZMC5M4IY5DF5IFZEMUBH7A7XQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.860656125 (263h19m27.819753375s) [noise=true , address=104.36.82.8:4970 , piece_id=UVCDEDYGRU2MRGARTHX6HE37MN7R7ENCKECID43OY5AKGRGUV4OA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:05.892818451 (263h19m27.851915686s) [noise=true , address=104.36.82.6:4874 , piece_id=QWVSGKI7TMJ4WANYEGXAEELGZ56MABCNHMMHB5BNRYQPMXWCZLPQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: <nil> Aug 19 16:45:34.599483064 (263h19m56.55858031s) [noise=true , address=77.90.31.19:4841 , piece_id=V4IQZZIOEUNRM2UZQM43LVAU7DD2Z6GCGXDV2ELDOA4SX7SRUYSQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:45:34.599584471 (263h19m56.558681702s) [noise=true , address=77.90.31.16:4735 , piece_id=4BINUM36RCVMJNSGOS2K2Y4V4NY3FYFTGE2LLLAMRZTI74YLAJ2A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 139.59.144.22:44230->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599644420 (263h19m56.558741652s) [noise=true , address=45.91.116.13:4154 , piece_id=EWEONRKBR5FWUKRYS3XKZVBKNXDNAZ7N7YTNFIB7N6Z4NHMM7UZQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: read tcp 139.59.144.22:49616->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599689167 (263h19m56.558786398s) [noise=true , address=45.91.116.9:4263 , piece_id=WRYOPKJBZPYLLBEX5NS23237O57QTFOHO73HG4QH5SWBCUHHGN6A, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33984->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599737297 (263h19m56.558834528s) [noise=true , address=45.91.116.7:4011 , piece_id=SCM2R7MZ7BXHKAC3HZJ2RO46AY256MAULWCCYWRROZK7GM2VSGHA, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: read tcp 139.59.144.22:39790->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599743407 (263h19m56.558840639s) [noise=true , address=45.91.116.11:4118 , piece_id=O4JFCMWX3MAMQVTF7MTCY33WLOH6SK6ODNK4BP5KZQJ22OJOIBGQ, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 139.59.144.22:53524->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599799239 (263h19m56.558896488s) [noise=true , address=77.90.31.18:4807 , piece_id=WQHNTCFW2YOSYYHNJ7HYTJ74PLJKIKXVTBUKGYMHN65LD7XFPC5Q, seg_pos={0 0}, upload=274142]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 139.59.144.22:54628->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:34.599857068 (263h19m56.558954325s) [seg_pos={0 0}, upload=274142]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:34.724663931 (263h19m56.683761189s) [upload=274142]: Done uploading segments... Aug 19 16:45:34.724702399 (263h19m56.683799654s) [upload=274142]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 2.558065513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.908885515s FAIL (2025-08-19 16:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382251427 (263h18m59.341348675s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382414984 (263h18m59.341512232s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382673030 (263h18m59.341770276s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382735719 (263h18m59.341832965s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383053107 (263h18m59.34215045s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383084717 (263h18m59.342181964s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383534576 (263h18m59.342631858s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383917134 (263h18m59.343014388s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384348854 (263h18m59.343446109s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384718383 (263h18m59.343815637s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385071761 (263h18m59.344169015s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385434046 (263h18m59.344531298s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385721240 (263h18m59.344818486s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.386090919 (263h18m59.345188282s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.964767860 (263h18m59.923865117s) [upload=274134]: Next returned nil segment Aug 19 16:44:37.964798911 (263h18m59.923896169s) [upload=274134]: Waiting for error group managing segments... Aug 19 16:44:38.110619970 (263h19m0.069717223s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.147656211 (263h19m0.106753456s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.151775591 (263h19m0.110872839s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.154994774 (263h19m0.114092024s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.157317429 (263h19m0.116414677s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.162078807 (263h19m0.121176058s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.184864439 (263h19m0.143961692s) [noise=true , address=104.36.82.39:4738 , piece_id=FBW64WJA6ARTNDKWRXCTJC7PK2WP5KRBUZ3EJVP3HVWTB5G7B5IA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.296041507 (263h19m0.255138762s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:39.007977043 (263h19m0.967074295s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:45:36.900803755 (263h19m58.859901013s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: read tcp 139.59.144.22:47708->77.90.31.15:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.900894139 (263h19m58.859991396s) [noise=true , address=45.91.116.8:4068 , piece_id=2CZMYUU6FSTFLXFMH7ZAW3XE4N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82, address:45.91.116.8:4068): protocol: read tcp 139.59.144.22:51570->45.91.116.8:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901206820 (263h19m58.860304076s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: write tcp 139.59.144.22:54138->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901449505 (263h19m58.860546763s) [noise=true , address=77.90.31.21:4886 , piece_id=6HGSVG3VGLNSZSFYKNR4KEUXWS366XVYRE5HCP45RGJTYCVZMZ5A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection Aug 19 16:45:36.901524916 (263h19m58.860622165s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: write tcp 139.59.144.22:34756->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901587712 (263h19m58.860684948s) [noise=true , address=45.91.116.11:4123 , piece_id=RK4R3IGJWRZMVCMKEBD3KCB4NPIM53MX34PCEL3MQSZRCS7I6T2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12kjHfRVsDASQz36zUAHrXsLJausE7UTNzHgiFWpys4bQK21cAE, address:45.91.116.11:4123): protocol: read tcp 139.59.144.22:52338->45.91.116.11:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901649530 (263h19m58.860746782s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:47534->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901713013 (263h19m58.860810272s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: read tcp 139.59.144.22:35448->77.90.31.11:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901775369 (263h19m58.860872604s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: read tcp 139.59.144.22:33368->77.90.31.10:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901796067 (263h19m58.860893328s) [seg_pos={0 0}, upload=274134]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 19 16:45:37.011173029 (263h19m58.970270282s) [upload=274134]: Done uploading segments... Aug 19 16:45:37.011204568 (263h19m58.970301824s) [upload=274134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.908929801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X....XXX...] duration: 8.200312481s FAIL (2025-08-19 16:46:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 8.200341809s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 28.475393213s FAIL (2025-08-19 16:46:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382251427 (263h18m59.341348675s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382414984 (263h18m59.341512232s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382673030 (263h18m59.341770276s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.382735719 (263h18m59.341832965s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383053107 (263h18m59.34215045s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383084717 (263h18m59.342181964s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383534576 (263h18m59.342631858s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.383917134 (263h18m59.343014388s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384348854 (263h18m59.343446109s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.384718383 (263h18m59.343815637s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385071761 (263h18m59.344169015s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385434046 (263h18m59.344531298s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.385721240 (263h18m59.344818486s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.386090919 (263h18m59.345188282s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Uploading piece... Aug 19 16:44:37.964767860 (263h18m59.923865117s) [upload=274134]: Next returned nil segment Aug 19 16:44:37.964798911 (263h18m59.923896169s) [upload=274134]: Waiting for error group managing segments... Aug 19 16:44:38.110619970 (263h19m0.069717223s) [noise=true , address=104.36.82.35:4614 , piece_id=IVDNMWG4UB527PG2I2UD5DT5O5TOPKBZWGXSCCP4IC32DG4JR4RQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.147656211 (263h19m0.106753456s) [noise=true , address=104.36.82.44:4879 , piece_id=PCVWIS7HFIWA7BMB42WWXNCL5MLBWXFUWO4U5UBW67BU7MYL67PQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.151775591 (263h19m0.110872839s) [noise=true , address=104.36.82.4:4814 , piece_id=NJQM6LY3V4Q55F6QKG52B4CECS5VNOVFC4IFSW4BNMKUHG66AZVQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.154994774 (263h19m0.114092024s) [noise=true , address=104.36.82.59:4304 , piece_id=NWUMLVDXE6PB3SUVDLNGAUIDQKK4C33VNEHJWVICK7HZG35NMJHA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.157317429 (263h19m0.116414677s) [noise=true , address=104.36.82.38:4699 , piece_id=4JZXELKX3XURHZE726FJBZIBKCH55KG22GFEKTLXNWI7IIGEKF2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.162078807 (263h19m0.121176058s) [noise=true , address=104.36.82.11:4125 , piece_id=CG34LDAB6C4C7PBUZDAHZLEYTAXHEMVXEW7LOQRRDJTF3TDLGFLQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.184864439 (263h19m0.143961692s) [noise=true , address=104.36.82.39:4738 , piece_id=FBW64WJA6ARTNDKWRXCTJC7PK2WP5KRBUZ3EJVP3HVWTB5G7B5IA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:38.296041507 (263h19m0.255138762s) [noise=true , address=79.127.220.98:4400 , piece_id=VBCZPWCP6DHFZTVM4QE4K4HSUTY2RZOOVZSSJZYO5CIFP3MZ3KTA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:44:39.007977043 (263h19m0.967074295s) [noise=true , address=104.36.82.5:4869 , piece_id=U7IKILG4KIQHEDJABLGMDEHVMWCEMLVNKD4YUADOL7AF2CYTSL2Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: <nil> Aug 19 16:45:36.900803755 (263h19m58.859901013s) [noise=true , address=77.90.31.15:4708 , piece_id=MKIINZ5BKFYA3H2AOIH6WEENMWUGHMFM3YOCPT4MV5AW3WYTBTYA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: read tcp 139.59.144.22:47708->77.90.31.15:4708: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.900894139 (263h19m58.859991396s) [noise=true , address=45.91.116.8:4068 , piece_id=2CZMYUU6FSTFLXFMH7ZAW3XE4N33FLEJZYSNEFPZ2KW2GNLMHQIQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82, address:45.91.116.8:4068): protocol: read tcp 139.59.144.22:51570->45.91.116.8:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901206820 (263h19m58.860304076s) [noise=true , address=45.91.116.7:4031 , piece_id=FMHK5ISVFEVT6U2VY7F5LQ2TC2QEHP2ZJ2JBG7B2PNXXEL7QRPPQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: write tcp 139.59.144.22:54138->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901449505 (263h19m58.860546763s) [noise=true , address=77.90.31.21:4886 , piece_id=6HGSVG3VGLNSZSFYKNR4KEUXWS366XVYRE5HCP45RGJTYCVZMZ5A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42172->77.90.31.21:4886: use of closed network connection Aug 19 16:45:36.901524916 (263h19m58.860622165s) [noise=true , address=45.91.116.9:4286 , piece_id=P2ASJUN2IEPMURP5Y5B5IVN33CHNONWQ2XYHYQD4DDQ2FLD4Q7OQ, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: write tcp 139.59.144.22:34756->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901587712 (263h19m58.860684948s) [noise=true , address=45.91.116.11:4123 , piece_id=RK4R3IGJWRZMVCMKEBD3KCB4NPIM53MX34PCEL3MQSZRCS7I6T2A, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12kjHfRVsDASQz36zUAHrXsLJausE7UTNzHgiFWpys4bQK21cAE, address:45.91.116.11:4123): protocol: read tcp 139.59.144.22:52338->45.91.116.11:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901649530 (263h19m58.860746782s) [noise=true , address=77.90.31.17:4777 , piece_id=3UJLGJ6HWUTZUPSAEE7EMMAI523L2WMHDI37A2P2E5MN6CIUCCDA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:47534->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901713013 (263h19m58.860810272s) [noise=true , address=77.90.31.11:4570 , piece_id=CTOQRW7ASAZ2UITQ7HWJ45SRTL35BLZVH4EUDLSGHU5QZSNYJNKA, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: read tcp 139.59.144.22:35448->77.90.31.11:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901775369 (263h19m58.860872604s) [noise=true , address=77.90.31.10:4552 , piece_id=RM6EF74VE5CCLYDETKJON5EVS7HP4QRKIYGF54M3TA7AWBNN553Q, seg_pos={0 0}, upload=274134]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: read tcp 139.59.144.22:33368->77.90.31.10:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:36.901796067 (263h19m58.860893328s) [seg_pos={0 0}, upload=274134]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 19 16:45:37.011173029 (263h19m58.970270282s) [upload=274134]: Done uploading segments... Aug 19 16:45:37.011204568 (263h19m58.970301824s) [upload=274134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 28.475497528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X............X.................................] duration: 1.459711099s FAIL (2025-08-19 16:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 0d35685df3a986652a34c4382db38e8e3160748c04f145b21040bb0e10da9778 != before 056e53e2ffdb8a7be351f4dd2196dc136d878b374889f8e4eb517331701c07a0 ======LOGS====== Aug 19 16:20:50.987835331 (262h55m12.946932571s) [upload=273716]: Starting upload Aug 19 16:20:50.987891887 (262h55m12.946989132s) [upload=273716]: Done starting upload Aug 19 16:20:50.987965614 (262h55m12.94706285s) [upload=273716]: Uploading segments... Aug 19 16:20:50.988001192 (262h55m12.947098446s) [upload=273716]: Got next segment. Inline: true Aug 19 16:20:50.988007924 (262h55m12.947105169s) [upload=273716]: Waiting for error group managing segments... Aug 19 16:20:50.988050561 (262h55m12.947147801s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:20:50.988055364 (262h55m12.947152609s) [upload=273716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:20:50.988059019 (262h55m12.947156264s) [upload=273716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:20:51.294533182 (262h55m13.253630433s) [upload=273716]: Done uploading segments... Aug 19 16:20:51.294559502 (262h55m13.253656758s) [upload=273716]: Upload finished. err: <nil> Duration:* 1.459760441s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)