Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 2.632742139s FAIL (2025-08-19 17:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.632771962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X..........X.X..............] duration: 1.030583395s FAIL (2025-08-19 17:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.030649938s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 2.65300947s FAIL (2025-08-19 17:13:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.653072106s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX...XX...............................] duration: 2.813953939s FAIL (2025-08-19 17:13:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813989148s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 2.336840485s FAIL (2025-08-19 17:12:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.336874463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..........X.X.............] duration: 1.045632137s FAIL (2025-08-19 17:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.045673975s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 2.518489358s FAIL (2025-08-19 17:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.518548102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....XXX.....................................] duration: 2.396775967s FAIL (2025-08-19 17:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.396814012s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 2.347214215s FAIL (2025-08-19 17:12:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.347248402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..........X.X............] duration: 922.975166ms FAIL (2025-08-19 17:12:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 923.006942ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 2.442555557s FAIL (2025-08-19 17:12:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.442580357s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX...XX..............................] duration: 2.714387277s FAIL (2025-08-19 17:12:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.714413647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....XXX....................................] duration: 2.395011348s FAIL (2025-08-19 17:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.395047499s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 2.442418415s FAIL (2025-08-19 17:11:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.442450434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 2.438356031s FAIL (2025-08-19 17:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.438385733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X....XXX...................................] duration: 2.599453308s FAIL (2025-08-19 17:11:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.599484596s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XX...XX.............................] duration: 2.863894411s FAIL (2025-08-19 17:11:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.86391932s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..........X.X...........] duration: 977.593241ms FAIL (2025-08-19 17:11:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 977.630182ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX...XX............................] duration: 2.75163675s FAIL (2025-08-19 17:10:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752045845s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 2.431113651s FAIL (2025-08-19 17:10:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.43114555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................X..........X.X..........] duration: 1.342217725s FAIL (2025-08-19 17:10:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.342265466s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X....XXX..................................] duration: 2.40663931s FAIL (2025-08-19 17:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.406669702s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 2.663549675s FAIL (2025-08-19 17:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.663580616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 2.554075695s FAIL (2025-08-19 17:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.554107317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....XXX.................................] duration: 2.446175731s FAIL (2025-08-19 17:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.446219239s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X..........X.X.........] duration: 1.179104787s FAIL (2025-08-19 17:09:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.179131741s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 2.53813587s FAIL (2025-08-19 17:09:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.538169877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX...XX...........................] duration: 3.034014275s FAIL (2025-08-19 17:09:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.034044306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 2.499050049s FAIL (2025-08-19 17:09:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.499087393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X....XXX................................] duration: 2.556340566s FAIL (2025-08-19 17:09:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.556382437s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................X..........X.X........] duration: 1.023587835s FAIL (2025-08-19 17:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.023622361s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....XXX...............................] duration: 2.769148838s FAIL (2025-08-19 17:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.769196155s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X......................................X..........X.X.......] duration: 864.662084ms FAIL (2025-08-19 17:08:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 864.698432ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX...XX..........................] duration: 2.715639282s FAIL (2025-08-19 17:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.715750896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X....XXX..............................] duration: 3.326600382s FAIL (2025-08-19 17:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 3.326631588s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 2.439647137s FAIL (2025-08-19 17:08:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.439680516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 10.800200583s FAIL (2025-08-19 17:08:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 10.800231502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....XXX.............................] duration: 2.441822583s FAIL (2025-08-19 17:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.441856434s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 2.745230169s FAIL (2025-08-19 17:07:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.745263636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 2.460059866s FAIL (2025-08-19 17:07:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.460090046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X......................................X..........X.X......] duration: 912.144155ms FAIL (2025-08-19 17:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 912.178515ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XX...XX.........................] duration: 3.010671775s FAIL (2025-08-19 17:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.010727738s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 2.366535343s FAIL (2025-08-19 17:07:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.366587035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X....XXX............................] duration: 2.538109322s FAIL (2025-08-19 17:07:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.53814944s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 2.597430574s FAIL (2025-08-19 17:07:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.597462331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X......................................X..........X.X.....] duration: 832.263634ms FAIL (2025-08-19 17:06:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 832.29307ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX...XX........................] duration: 2.751821029s FAIL (2025-08-19 17:06:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.751872733s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 2.409474324s FAIL (2025-08-19 17:06:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 5.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233115087 (263h22m59.192212323s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.233528098 (263h22m59.192625335s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.234408322 (263h22m59.193505553s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235061500 (263h22m59.19415874s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.235710927 (263h22m59.194808161s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236305251 (263h22m59.195402499s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.236804674 (263h22m59.195901911s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.238972618 (263h22m59.198069926s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.239892336 (263h22m59.198989573s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.240529893 (263h22m59.199627132s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.241129861 (263h22m59.2002271s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.242937842 (263h22m59.202035082s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.244910507 (263h22m59.204007741s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.246657050 (263h22m59.205754289s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.248689412 (263h22m59.207786647s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Uploading piece... Aug 19 16:48:37.891525132 (263h22m59.850622383s) [upload=274202]: Next returned nil segment Aug 19 16:48:37.891576150 (263h22m59.850673401s) [upload=274202]: Waiting for error group managing segments... Aug 19 16:48:38.020617556 (263h22m59.979714789s) [noise=true , address=104.36.82.9:4007 , piece_id=CCYMNVXXN2KBQHBXN44B27W3RCPWDT6FD7XOGDKMBZ3IVNWPPMUA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.026650569 (263h22m59.985747815s) [noise=true , address=104.36.82.6:4875 , piece_id=IKNKTXQUXTEALXVPO5PX6266T27HZVDFVH72SOD5774567U7WJVA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.027486751 (263h22m59.986583998s) [noise=true , address=104.36.82.11:4114 , piece_id=NSQXF7ALDSI6SQZN2K5OIH2VQJNGCYQU32WBZQAH4MKG5LD3DJHQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.032660169 (263h22m59.991757415s) [noise=true , address=104.36.82.11:4113 , piece_id=BIZXPM5GSE35NLEFPIHMFHELBM2V45PLMR2FNZ4D3VPFUHVDRKRQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.036528128 (263h22m59.99562536s) [noise=true , address=104.36.82.16:4295 , piece_id=VGOKYGO4PJYEKTVICVAAZ7ORNCVFI37PL2G43JORV5GNC2DBSK5Q, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:48:38.075741564 (263h23m0.034838804s) [noise=true , address=104.36.82.9:4006 , piece_id=QYXJTRPXTDZV2V5XXDQKTTVBX4QIQFVUJGLHIH4ZAI3PUS7ILVDA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: <nil> Aug 19 16:49:06.841310322 (263h23m28.800407559s) [noise=true , address=45.91.116.11:4117 , piece_id=6UINCPDHVODQ4BOMYXBOAEVUL5WD6AGPTLK4FPPHCLWIJBL2VLJA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34108->45.91.116.11:4117: use of closed network connection; read tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; close tcp 139.59.144.22:34092->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841493363 (263h23m28.80059061s) [noise=true , address=45.91.116.4:4940 , piece_id=OG2PWCWXXDQHMOKL76SI4WCOBMP6KC774BZLX6CRJXTVD52QEHKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841710282 (263h23m28.800807515s) [noise=true , address=77.90.31.19:4841 , piece_id=QR63HXXKMA77T4UI6VRURW4NDTZVSOQ33JEVRGNJVQKKO6OY6PCQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.841957265 (263h23m28.801054502s) [noise=true , address=77.90.31.19:4842 , piece_id=43BY54IQEVWSMYYK2SJEHSANMZUQOF3I235AKAKHTNJN7VH4BJWA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:42958->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842015441 (263h23m28.801112678s) [noise=true , address=77.90.31.12:4590 , piece_id=QKSSMVQ4TZ54EIPW7MBOJBABFN5WETZB6UJTIXPNOHEOU5R35VEQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:34282->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842193782 (263h23m28.801291028s) [noise=true , address=45.91.116.10:4081 , piece_id=OOVMV4HD7FWDGOFNNMZWSTF6Q5BGAYXGOYQMEFFXXRCAAV4CMHNQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842215651 (263h23m28.801312886s) [noise=true , address=77.90.31.11:4555 , piece_id=NLGBKVFDPY5GZPM7Y6NLCFAG677654IHJOQ7OJX4XAGY5XFCGK6A, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 139.59.144.22:51644->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842309067 (263h23m28.801406306s) [noise=true , address=45.91.116.7:4011 , piece_id=4XNUMLPLFQQKHMBYPLTLKMLSQULLPLJ745YSMCFQLKO74SDT6DSA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842375662 (263h23m28.801472899s) [noise=true , address=45.91.116.9:4263 , piece_id=BSH4RFWHDOII6EBPVJPWLFR7CSCSALEO6JX2GO2G5OAAHHVVBEKQ, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:48978->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842394128 (263h23m28.801491364s) [noise=true , address=45.91.116.6:4973 , piece_id=M5SB5WLCHHO3TK4ET757MFGY2OGPDMZXYUHCWL5QH33AQBIWT2DA, seg_pos={0 0}, upload=274202]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:06.842444119 (263h23m28.801541372s) [seg_pos={0 0}, upload=274202]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 16:49:06.945306874 (263h23m28.904404121s) [upload=274202]: Done uploading segments... Aug 19 16:49:06.945341006 (263h23m28.904438254s) [upload=274202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.409510283s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX...XX.......................] duration: 3.530346944s FAIL (2025-08-19 17:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530443221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 2.522215082s FAIL (2025-08-19 17:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.522250381s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)