Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX.] duration: 2.555302054s FAIL (2025-07-26 23:47:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.55533152s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......X........XX....X....X.X..X........XX.X....X......X..X] duration: 1m0.000695062s FAIL (2025-07-26 23:47:11 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000724629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...] duration: 43.985990765s FAIL (2025-07-26 23:47:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 386477011s) [upload=57629]: Done uploading segments... Jul 26 23:42:36.196198115 (60h7m7.386496688s) [upload=57629]: Upload finished. err: <nil> Jul 26 23:42:53.417482923 (60h7m24.607781508s) [upload=57646]: Starting upload Jul 26 23:42:53.417579840 (60h7m24.607878408s) [upload=57646]: Done starting upload Jul 26 23:42:53.417683172 (60h7m24.607981743s) [upload=57646]: Uploading segments... Jul 26 23:42:53.417899571 (60h7m24.608198145s) [upload=57646]: Got next segment. Inline: false Jul 26 23:42:53.417974540 (60h7m24.608273123s) [upload=57646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:53.417980468 (60h7m24.608279048s) [upload=57646]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:53.619456625 (60h7m24.809755192s) [seg_pos={0 0}, upload=57646]: Begin upload segment... Jul 26 23:42:53.619510776 (60h7m24.809809355s) [seg_pos={0 0}, upload=57646]: Done begin upload segment. Jul 26 23:42:53.619533396 (60h7m24.809831979s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620054445 (60h7m24.810353026s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620302982 (60h7m24.810601565s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620601932 (60h7m24.810900518s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621053086 (60h7m24.811351675s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621414282 (60h7m24.811712867s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621692545 (60h7m24.811991124s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622057938 (60h7m24.812356524s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622416760 (60h7m24.812715343s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622854791 (60h7m24.813155765s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623164302 (60h7m24.813462881s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623692679 (60h7m24.813991267s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624035038 (60h7m24.814333618s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624380099 (60h7m24.81467869s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624733702 (60h7m24.815032284s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625162941 (60h7m24.815461531s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625502379 (60h7m24.815800961s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625734416 (60h7m24.816032999s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:54.036062824 (60h7m25.226361413s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:54.190797058 (60h7m25.381095651s) [upload=57646]: Next returned nil segment Jul 26 23:42:54.190822148 (60h7m25.381120738s) [upload=57646]: Waiting for error group managing segments... Jul 26 23:42:54.375177273 (60h7m25.565475845s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.382307219 (60h7m25.572605792s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.908069114 (60h7m26.098367696s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.054218318 (60h7m26.244516891s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.134081796 (60h7m26.324380368s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:59.514001760 (60h7m30.70430033s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:00.521755754 (60h7m31.712054344s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:03.673252172 (60h7m34.863550755s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:13.898059806 (60h7m45.088358376s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:22.289157029 (60h7m53.479455602s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:37.181386006 (60h8m8.37168459s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:40.826146516 (60h8m12.016445106s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:42.389900272 (60h8m13.580198844s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:43.133590638 (60h8m14.323889205s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:53.139632762 (60h8m24.329931335s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139757243 (60h8m24.330055823s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139822221 (60h8m24.33012079s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: read tcp 139.59.144.22:38110->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:53.139855040 (60h8m24.330153626s) [seg_pos={0 0}, upload=57646]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:43:53.247153679 (60h8m24.437452249s) [upload=57646]: Done uploading segments... Jul 26 23:43:53.247179655 (60h8m24.437478234s) [upload=57646]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 43.986085933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [.X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.] duration: 30.002408187s FAIL (2025-07-26 23:46:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 5:14.939323205 (60h9m46.129621778s) [upload=57688]: Uploading segments... Jul 26 23:45:14.939715296 (60h9m46.130013862s) [upload=57688]: Got next segment. Inline: false Jul 26 23:45:14.939809688 (60h9m46.13010826s) [upload=57688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:14.939816721 (60h9m46.1301153s) [upload=57688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:15.049115070 (60h9m46.23941364s) [seg_pos={0 0}, upload=57688]: Begin upload segment... Jul 26 23:45:15.049177624 (60h9m46.2394762s) [seg_pos={0 0}, upload=57688]: Done begin upload segment. Jul 26 23:45:15.049206397 (60h9m46.239504985s) [noise=true , address=77.90.31.19:4843 , piece_id=NV4POVP7DMB6U2QP7FPX66JODUY25APMQQ4XS3HHXE6627CBOUQA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049253816 (60h9m46.239552412s) [noise=true , address=45.91.116.13:4153 , piece_id=BNIL73IAQFMCSVM7JABK5WWCQN5AAWCE6AA7XHAXYWF5SRAKN4RQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049656868 (60h9m46.239955453s) [noise=true , address=77.90.31.16:4733 , piece_id=U7TNWGWKFTKEFBECQMHLHSFKVHMBOJKOB5AAW6OQZZF7MBNFO53A, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049795623 (60h9m46.240094203s) [noise=true , address=45.91.116.5:4299 , piece_id=4SPIXSDTNFQFLI2V264ZAFKV6HA3JDF3C773DKTBYRXOCO76PBHQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.050081914 (60h9m46.240380497s) [noise=true , address=45.91.116.4:4939 , piece_id=6GQ6Y2Q2CEWOTT76VGUOCEBREMIQM3EKDOZSXPR7YLT3GGQWFC3Q, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.050412949 (60h9m46.240711531s) [noise=true , address=104.36.82.5:4838 , piece_id=LK5PQ2CQRGZHGJXSHN3WMG2NPHQBP5LOBCGJUQBYAB6YJVNYQ3KQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.052066885 (60h9m46.242365471s) [noise=true , address=45.91.116.7:4010 , piece_id=YIG5YFYFUXLA5NDX3SF7POXA3QMTASC6RUPSZ4HW23FYLGBYFLKQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.052461082 (60h9m46.242759669s) [noise=true , address=104.36.82.15:4259 , piece_id=TFXRFZF5MN23PQQVKEORMHGAJTKBZ5RSP7X4ZCKZOZM4ZPUR36AQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.053256381 (60h9m46.243554966s) [noise=true , address=104.36.82.9:4006 , piece_id=MUNLNOD3GC3FSYTSFFCOAGG4TGUJUV2R2F5R7WIQ73Q3GZBGVP6Q, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.053862801 (60h9m46.244161388s) [noise=true , address=77.90.31.14:4661 , piece_id=QV4SDJ7WSDQ75KJEI6CNNVUE24RQS2ESQXP2JXQJWOMKID4B74QA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.054239661 (60h9m46.244538259s) [noise=true , address=77.90.31.13:4626 , piece_id=64KNS5OBIIGLIDYQYLRWY3G6JE6P4XJGF5THHVEKEO7273LG2TJA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.054639584 (60h9m46.24493817s) [noise=true , address=45.91.116.12:4225 , piece_id=36HDQQF3GS6EAB5RJT57S4E6QRQM33VERJ62L4JR44CWNIZUZSAA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055045568 (60h9m46.245344157s) [noise=true , address=45.91.116.9:4263 , piece_id=VW732EWMNBO2OYC4BX4WW3DLCO63UIO7JYSBOQXSVT4QZI5RXODQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055187348 (60h9m46.245485927s) [noise=true , address=77.90.31.14:4662 , piece_id=WD4ZWKMRQ3QFTGU5FOMSXG3ZEZ4ULDBVUWESRY65K5KS27DGTVGA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055603874 (60h9m46.245902458s) [noise=true , address=104.36.82.4:4803 , piece_id=YDHI53FPM6ARYWCOJJ7JV4VB3EG5NXCZEOG75NLOAVMYIQOUR2KA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055999800 (60h9m46.246298388s) [noise=true , address=104.36.82.13:4186 , piece_id=KNQHWOH4KF66CF4S43WAD2CIQ6IPB2WRBZBKWLY7YMN5E5QB7WZA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.653271887 (60h9m46.843570458s) [upload=57688]: Next returned nil segment Jul 26 23:45:15.653289829 (60h9m46.8435884s) [upload=57688]: Waiting for error group managing segments... Jul 26 23:45:15.795342193 (60h9m46.985640777s) [noise=true , address=77.90.31.14:4661 , piece_id=QV4SDJ7WSDQ75KJEI6CNNVUE24RQS2ESQXP2JXQJWOMKID4B74QA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.795885584 (60h9m46.986184173s) [noise=true , address=45.91.116.4:4939 , piece_id=6GQ6Y2Q2CEWOTT76VGUOCEBREMIQM3EKDOZSXPR7YLT3GGQWFC3Q, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.796644370 (60h9m46.98694296s) [noise=true , address=45.91.116.7:4010 , piece_id=YIG5YFYFUXLA5NDX3SF7POXA3QMTASC6RUPSZ4HW23FYLGBYFLKQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.798564259 (60h9m46.988862841s) [noise=true , address=45.91.116.13:4153 , piece_id=BNIL73IAQFMCSVM7JABK5WWCQN5AAWCE6AA7XHAXYWF5SRAKN4RQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.810614979 (60h9m47.000913566s) [noise=true , address=45.91.116.5:4299 , piece_id=4SPIXSDTNFQFLI2V264ZAFKV6HA3JDF3C773DKTBYRXOCO76PBHQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.816720332 (60h9m47.0070189s) [noise=true , address=77.90.31.14:4662 , piece_id=WD4ZWKMRQ3QFTGU5FOMSXG3ZEZ4ULDBVUWESRY65K5KS27DGTVGA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.817110694 (60h9m47.007409266s) [noise=true , address=45.91.116.12:4225 , piece_id=36HDQQF3GS6EAB5RJT57S4E6QRQM33VERJ62L4JR44CWNIZUZSAA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:16.276598306 (60h9m47.466896877s) [noise=true , address=45.91.116.9:4263 , piece_id=VW732EWMNBO2OYC4BX4WW3DLCO63UIO7JYSBOQXSVT4QZI5RXODQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:17.179249485 (60h9m48.369548057s) [noise=true , address=77.90.31.13:4626 , piece_id=64KNS5OBIIGLIDYQYLRWY3G6JE6P4XJGF5THHVEKEO7273LG2TJA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:17.961185907 (60h9m49.151484487s) [noise=true , address=77.90.31.19:4843 , piece_id=NV4POVP7DMB6U2QP7FPX66JODUY25APMQQ4XS3HHXE6627CBOUQA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:44.671837666 (60h10m15.862136232s) [noise=true , address=104.36.82.9:4006 , piece_id=MUNLNOD3GC3FSYTSFFCOAGG4TGUJUV2R2F5R7WIQ73Q3GZBGVP6Q, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection Jul 26 23:45:44.671882030 (60h10m15.8621806s) [noise=true , address=104.36.82.5:4838 , piece_id=LK5PQ2CQRGZHGJXSHN3WMG2NPHQBP5LOBCGJUQBYAB6YJVNYQ3KQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection; write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection Jul 26 23:45:44.671915818 (60h10m15.862214383s) [noise=true , address=104.36.82.4:4803 , piece_id=YDHI53FPM6ARYWCOJJ7JV4VB3EG5NXCZEOG75NLOAVMYIQOUR2KA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection Jul 26 23:45:44.671951201 (60h10m15.862249771s) [noise=true , address=77.90.31.16:4733 , piece_id=U7TNWGWKFTKEFBECQMHLHSFKVHMBOJKOB5AAW6OQZZF7MBNFO53A, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:44.671972905 (60h10m15.86227147s) [noise=true , address=104.36.82.15:4259 , piece_id=TFXRFZF5MN23PQQVKEORMHGAJTKBZ5RSP7X4ZCKZOZM4ZPUR36AQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection; write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection Jul 26 23:45:44.672030900 (60h10m15.862329469s) [noise=true , address=104.36.82.13:4186 , piece_id=KNQHWOH4KF66CF4S43WAD2CIQ6IPB2WRBZBKWLY7YMN5E5QB7WZA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:47862->104.36.82.13:4186: use of closed network connection; write tcp 139.59.144.22:47862->104.36.82.13:4186: use of closed network connection Jul 26 23:45:44.672062958 (60h10m15.86236155s) [seg_pos={0 0}, upload=57688]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:45:44.776599211 (60h10m15.966897803s) [upload=57688]: Done uploading segments... Jul 26 23:45:44.776635025 (60h10m15.966933616s) [upload=57688]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.002436326s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX....XX] duration: 30.001566807s FAIL (2025-07-26 23:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001642255s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX] duration: 30.112172997s FAIL (2025-07-26 23:46:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.112188082s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX....X] duration: 30.001557815s FAIL (2025-07-26 23:46:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001591075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXX] duration: 30.114778827s FAIL (2025-07-26 23:46:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.114824783s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.......X........XX....X....X.X..X........XX.X....X......X..] duration: 47.435646336s FAIL (2025-07-26 23:46:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.435675008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X..] duration: 40.657095809s FAIL (2025-07-26 23:45:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 386477011s) [upload=57629]: Done uploading segments... Jul 26 23:42:36.196198115 (60h7m7.386496688s) [upload=57629]: Upload finished. err: <nil> Jul 26 23:42:53.417482923 (60h7m24.607781508s) [upload=57646]: Starting upload Jul 26 23:42:53.417579840 (60h7m24.607878408s) [upload=57646]: Done starting upload Jul 26 23:42:53.417683172 (60h7m24.607981743s) [upload=57646]: Uploading segments... Jul 26 23:42:53.417899571 (60h7m24.608198145s) [upload=57646]: Got next segment. Inline: false Jul 26 23:42:53.417974540 (60h7m24.608273123s) [upload=57646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:53.417980468 (60h7m24.608279048s) [upload=57646]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:53.619456625 (60h7m24.809755192s) [seg_pos={0 0}, upload=57646]: Begin upload segment... Jul 26 23:42:53.619510776 (60h7m24.809809355s) [seg_pos={0 0}, upload=57646]: Done begin upload segment. Jul 26 23:42:53.619533396 (60h7m24.809831979s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620054445 (60h7m24.810353026s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620302982 (60h7m24.810601565s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620601932 (60h7m24.810900518s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621053086 (60h7m24.811351675s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621414282 (60h7m24.811712867s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621692545 (60h7m24.811991124s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622057938 (60h7m24.812356524s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622416760 (60h7m24.812715343s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622854791 (60h7m24.813155765s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623164302 (60h7m24.813462881s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623692679 (60h7m24.813991267s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624035038 (60h7m24.814333618s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624380099 (60h7m24.81467869s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624733702 (60h7m24.815032284s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625162941 (60h7m24.815461531s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625502379 (60h7m24.815800961s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625734416 (60h7m24.816032999s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:54.036062824 (60h7m25.226361413s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:54.190797058 (60h7m25.381095651s) [upload=57646]: Next returned nil segment Jul 26 23:42:54.190822148 (60h7m25.381120738s) [upload=57646]: Waiting for error group managing segments... Jul 26 23:42:54.375177273 (60h7m25.565475845s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.382307219 (60h7m25.572605792s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.908069114 (60h7m26.098367696s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.054218318 (60h7m26.244516891s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.134081796 (60h7m26.324380368s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:59.514001760 (60h7m30.70430033s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:00.521755754 (60h7m31.712054344s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:03.673252172 (60h7m34.863550755s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:13.898059806 (60h7m45.088358376s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:22.289157029 (60h7m53.479455602s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:37.181386006 (60h8m8.37168459s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:40.826146516 (60h8m12.016445106s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:42.389900272 (60h8m13.580198844s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:43.133590638 (60h8m14.323889205s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:53.139632762 (60h8m24.329931335s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139757243 (60h8m24.330055823s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139822221 (60h8m24.33012079s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: read tcp 139.59.144.22:38110->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:53.139855040 (60h8m24.330153626s) [seg_pos={0 0}, upload=57646]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:43:53.247153679 (60h8m24.437452249s) [upload=57646]: Done uploading segments... Jul 26 23:43:53.247179655 (60h8m24.437478234s) [upload=57646]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 40.657139574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [..X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX] duration: 30.10662171s FAIL (2025-07-26 23:45:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 5:14.939323205 (60h9m46.129621778s) [upload=57688]: Uploading segments... Jul 26 23:45:14.939715296 (60h9m46.130013862s) [upload=57688]: Got next segment. Inline: false Jul 26 23:45:14.939809688 (60h9m46.13010826s) [upload=57688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:14.939816721 (60h9m46.1301153s) [upload=57688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:15.049115070 (60h9m46.23941364s) [seg_pos={0 0}, upload=57688]: Begin upload segment... Jul 26 23:45:15.049177624 (60h9m46.2394762s) [seg_pos={0 0}, upload=57688]: Done begin upload segment. Jul 26 23:45:15.049206397 (60h9m46.239504985s) [noise=true , address=77.90.31.19:4843 , piece_id=NV4POVP7DMB6U2QP7FPX66JODUY25APMQQ4XS3HHXE6627CBOUQA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049253816 (60h9m46.239552412s) [noise=true , address=45.91.116.13:4153 , piece_id=BNIL73IAQFMCSVM7JABK5WWCQN5AAWCE6AA7XHAXYWF5SRAKN4RQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049656868 (60h9m46.239955453s) [noise=true , address=77.90.31.16:4733 , piece_id=U7TNWGWKFTKEFBECQMHLHSFKVHMBOJKOB5AAW6OQZZF7MBNFO53A, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.049795623 (60h9m46.240094203s) [noise=true , address=45.91.116.5:4299 , piece_id=4SPIXSDTNFQFLI2V264ZAFKV6HA3JDF3C773DKTBYRXOCO76PBHQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.050081914 (60h9m46.240380497s) [noise=true , address=45.91.116.4:4939 , piece_id=6GQ6Y2Q2CEWOTT76VGUOCEBREMIQM3EKDOZSXPR7YLT3GGQWFC3Q, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.050412949 (60h9m46.240711531s) [noise=true , address=104.36.82.5:4838 , piece_id=LK5PQ2CQRGZHGJXSHN3WMG2NPHQBP5LOBCGJUQBYAB6YJVNYQ3KQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.052066885 (60h9m46.242365471s) [noise=true , address=45.91.116.7:4010 , piece_id=YIG5YFYFUXLA5NDX3SF7POXA3QMTASC6RUPSZ4HW23FYLGBYFLKQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.052461082 (60h9m46.242759669s) [noise=true , address=104.36.82.15:4259 , piece_id=TFXRFZF5MN23PQQVKEORMHGAJTKBZ5RSP7X4ZCKZOZM4ZPUR36AQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.053256381 (60h9m46.243554966s) [noise=true , address=104.36.82.9:4006 , piece_id=MUNLNOD3GC3FSYTSFFCOAGG4TGUJUV2R2F5R7WIQ73Q3GZBGVP6Q, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.053862801 (60h9m46.244161388s) [noise=true , address=77.90.31.14:4661 , piece_id=QV4SDJ7WSDQ75KJEI6CNNVUE24RQS2ESQXP2JXQJWOMKID4B74QA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.054239661 (60h9m46.244538259s) [noise=true , address=77.90.31.13:4626 , piece_id=64KNS5OBIIGLIDYQYLRWY3G6JE6P4XJGF5THHVEKEO7273LG2TJA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.054639584 (60h9m46.24493817s) [noise=true , address=45.91.116.12:4225 , piece_id=36HDQQF3GS6EAB5RJT57S4E6QRQM33VERJ62L4JR44CWNIZUZSAA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055045568 (60h9m46.245344157s) [noise=true , address=45.91.116.9:4263 , piece_id=VW732EWMNBO2OYC4BX4WW3DLCO63UIO7JYSBOQXSVT4QZI5RXODQ, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055187348 (60h9m46.245485927s) [noise=true , address=77.90.31.14:4662 , piece_id=WD4ZWKMRQ3QFTGU5FOMSXG3ZEZ4ULDBVUWESRY65K5KS27DGTVGA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055603874 (60h9m46.245902458s) [noise=true , address=104.36.82.4:4803 , piece_id=YDHI53FPM6ARYWCOJJ7JV4VB3EG5NXCZEOG75NLOAVMYIQOUR2KA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.055999800 (60h9m46.246298388s) [noise=true , address=104.36.82.13:4186 , piece_id=KNQHWOH4KF66CF4S43WAD2CIQ6IPB2WRBZBKWLY7YMN5E5QB7WZA, seg_pos={0 0}, upload=57688]: Uploading piece... Jul 26 23:45:15.653271887 (60h9m46.843570458s) [upload=57688]: Next returned nil segment Jul 26 23:45:15.653289829 (60h9m46.8435884s) [upload=57688]: Waiting for error group managing segments... Jul 26 23:45:15.795342193 (60h9m46.985640777s) [noise=true , address=77.90.31.14:4661 , piece_id=QV4SDJ7WSDQ75KJEI6CNNVUE24RQS2ESQXP2JXQJWOMKID4B74QA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.795885584 (60h9m46.986184173s) [noise=true , address=45.91.116.4:4939 , piece_id=6GQ6Y2Q2CEWOTT76VGUOCEBREMIQM3EKDOZSXPR7YLT3GGQWFC3Q, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.796644370 (60h9m46.98694296s) [noise=true , address=45.91.116.7:4010 , piece_id=YIG5YFYFUXLA5NDX3SF7POXA3QMTASC6RUPSZ4HW23FYLGBYFLKQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.798564259 (60h9m46.988862841s) [noise=true , address=45.91.116.13:4153 , piece_id=BNIL73IAQFMCSVM7JABK5WWCQN5AAWCE6AA7XHAXYWF5SRAKN4RQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.810614979 (60h9m47.000913566s) [noise=true , address=45.91.116.5:4299 , piece_id=4SPIXSDTNFQFLI2V264ZAFKV6HA3JDF3C773DKTBYRXOCO76PBHQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.816720332 (60h9m47.0070189s) [noise=true , address=77.90.31.14:4662 , piece_id=WD4ZWKMRQ3QFTGU5FOMSXG3ZEZ4ULDBVUWESRY65K5KS27DGTVGA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:15.817110694 (60h9m47.007409266s) [noise=true , address=45.91.116.12:4225 , piece_id=36HDQQF3GS6EAB5RJT57S4E6QRQM33VERJ62L4JR44CWNIZUZSAA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:16.276598306 (60h9m47.466896877s) [noise=true , address=45.91.116.9:4263 , piece_id=VW732EWMNBO2OYC4BX4WW3DLCO63UIO7JYSBOQXSVT4QZI5RXODQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:17.179249485 (60h9m48.369548057s) [noise=true , address=77.90.31.13:4626 , piece_id=64KNS5OBIIGLIDYQYLRWY3G6JE6P4XJGF5THHVEKEO7273LG2TJA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:17.961185907 (60h9m49.151484487s) [noise=true , address=77.90.31.19:4843 , piece_id=NV4POVP7DMB6U2QP7FPX66JODUY25APMQQ4XS3HHXE6627CBOUQA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: <nil> Jul 26 23:45:44.671837666 (60h10m15.862136232s) [noise=true , address=104.36.82.9:4006 , piece_id=MUNLNOD3GC3FSYTSFFCOAGG4TGUJUV2R2F5R7WIQ73Q3GZBGVP6Q, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50856->104.36.82.9:4006: use of closed network connection Jul 26 23:45:44.671882030 (60h10m15.8621806s) [noise=true , address=104.36.82.5:4838 , piece_id=LK5PQ2CQRGZHGJXSHN3WMG2NPHQBP5LOBCGJUQBYAB6YJVNYQ3KQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection; write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53332->104.36.82.5:4838: use of closed network connection Jul 26 23:45:44.671915818 (60h10m15.862214383s) [noise=true , address=104.36.82.4:4803 , piece_id=YDHI53FPM6ARYWCOJJ7JV4VB3EG5NXCZEOG75NLOAVMYIQOUR2KA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39484->104.36.82.4:4803: use of closed network connection Jul 26 23:45:44.671951201 (60h10m15.862249771s) [noise=true , address=77.90.31.16:4733 , piece_id=U7TNWGWKFTKEFBECQMHLHSFKVHMBOJKOB5AAW6OQZZF7MBNFO53A, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:44.671972905 (60h10m15.86227147s) [noise=true , address=104.36.82.15:4259 , piece_id=TFXRFZF5MN23PQQVKEORMHGAJTKBZ5RSP7X4ZCKZOZM4ZPUR36AQ, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection; write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45160->104.36.82.15:4259: use of closed network connection Jul 26 23:45:44.672030900 (60h10m15.862329469s) [noise=true , address=104.36.82.13:4186 , piece_id=KNQHWOH4KF66CF4S43WAD2CIQ6IPB2WRBZBKWLY7YMN5E5QB7WZA, seg_pos={0 0}, upload=57688]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:47862->104.36.82.13:4186: use of closed network connection; write tcp 139.59.144.22:47862->104.36.82.13:4186: use of closed network connection Jul 26 23:45:44.672062958 (60h10m15.86236155s) [seg_pos={0 0}, upload=57688]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:45:44.776599211 (60h10m15.966897803s) [upload=57688]: Done uploading segments... Jul 26 23:45:44.776635025 (60h10m15.966933616s) [upload=57688]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.106648464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [.X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXX] duration: 30.001598871s FAIL (2025-07-26 23:45:40 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.001629342s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X.......X........XX....X....X.X..X........XX.X....X......X.] duration: 32.289792345s FAIL (2025-07-26 23:45:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.289826699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX....] duration: 3.068517662s FAIL (2025-07-26 23:45:14 UTC) orbiter warning: context deadline exceeded Duration: 3.068555263s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [...X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXX] duration: 30.113096767s FAIL (2025-07-26 23:44:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ing upload Jul 26 23:44:28.307268823 (60h8m59.497567408s) [upload=57673]: Done starting upload Jul 26 23:44:28.307505082 (60h8m59.497803655s) [upload=57673]: Uploading segments... Jul 26 23:44:28.307618057 (60h8m59.497916629s) [upload=57673]: Got next segment. Inline: false Jul 26 23:44:28.307701569 (60h8m59.498000155s) [upload=57673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:44:28.307709126 (60h8m59.498007711s) [upload=57673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:44:28.440135695 (60h8m59.630434265s) [seg_pos={0 0}, upload=57673]: Begin upload segment... Jul 26 23:44:28.440182194 (60h8m59.630480776s) [seg_pos={0 0}, upload=57673]: Done begin upload segment. Jul 26 23:44:28.440213382 (60h8m59.630511964s) [noise=true , address=45.91.116.12:4225 , piece_id=ZLHXC67OYXE4LHEOGPKYUDV6TMYVZNXXDN7LSBMGJHUEN5DIQXDA, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.440249052 (60h8m59.630547636s) [noise=true , address=104.36.82.14:4221 , piece_id=ED6BSVEMDN3MYEDJ45ZWEJQMI23OUI4ROW24CY2TDT4U6YJXEIWQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.440620008 (60h8m59.630918592s) [noise=true , address=104.36.82.13:4187 , piece_id=ESQ3FJOKTAVQS5ICEFYK5YX3OACI33MWLB2WB6V7UKHIUCMNUQAQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.440946838 (60h8m59.631245424s) [noise=true , address=104.36.82.12:4149 , piece_id=2NNDBXJKQMJ7WIGMDESL5WYB3BLA5PZE6XKOSLYI53OC6NAXKWEQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.441045984 (60h8m59.631344575s) [noise=true , address=104.36.82.13:4185 , piece_id=EIIBBWVVPXNFY3HYF564U6DRELZRNO3CIOBFK2D3PUHGVMGPWBSQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.441357548 (60h8m59.63165613s) [noise=true , address=77.90.31.13:4626 , piece_id=UJRAAHF5YIQQPN2BMAY2ATCWLPHLXGSOLOEWA43AFAOKV5TZBRSQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.441508865 (60h8m59.631807455s) [noise=true , address=45.91.116.8:4047 , piece_id=G4OOGIFPHZ7NYPJLRRILNUWRNHDJBISCCPH54VOJS2B7VSKRHY6Q, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.441648592 (60h8m59.631947173s) [noise=true , address=45.91.116.10:4083 , piece_id=236G76NGG5I7QDZFZWSPHROVFVHU7VDO6G2QDVZSJKMVP2CLS75Q, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.441935789 (60h8m59.632234368s) [noise=true , address=104.36.82.2:4732 , piece_id=62UIZV5JB6UWLSWLN6JFK5P7VCJ4KB6VFUP6UZ4GCV2GNSL74EUQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442023435 (60h8m59.632322018s) [noise=true , address=77.90.31.13:4628 , piece_id=XXXBK35EZJQMBRGYOL2NQXWJT3YKHRSF5GBY5CWIXRK72TPUDO6A, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442285940 (60h8m59.632584505s) [noise=true , address=104.36.82.11:4115 , piece_id=OUDFQWGXQJOLL5YZ74TCZK3VO3VJDMPG55WKXXHBB7FRTD5PZHXQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442330854 (60h8m59.632629443s) [noise=true , address=45.91.116.4:4937 , piece_id=IA6WH3XERNH243EI3MUD52NW5Y2SUKXOST75OXGKTJHUFQR65GQQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442522208 (60h8m59.632820774s) [noise=true , address=45.91.116.5:4298 , piece_id=KSQG47Y77JZOXKTEABFKFCCSYPLFX3AB5Q7Y7IZYPQ76YEDNN4WQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442744334 (60h8m59.633042913s) [noise=true , address=45.91.116.11:4119 , piece_id=UWEB6A6U5QZT4FPVOSDLK2FADGE2ZLDUA6OZ6BU3ZX7DJCO22JTQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.442820312 (60h8m59.633118897s) [noise=true , address=45.91.116.6:4974 , piece_id=5JVTHPKIOZ6462BO7YGVQT2KATJTFCMJT327NJ5IGF5C44GTLPUQ, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:28.443237223 (60h8m59.633535812s) [noise=true , address=45.91.116.8:4048 , piece_id=VUMHPKQ3WBAY3VENB3IURNTWNGTNCTWITZGTGCDQD6B3A374JF6A, seg_pos={0 0}, upload=57673]: Uploading piece... Jul 26 23:44:29.088321065 (60h9m0.278619635s) [upload=57673]: Next returned nil segment Jul 26 23:44:29.088338797 (60h9m0.278637368s) [upload=57673]: Waiting for error group managing segments... Jul 26 23:44:29.194236022 (60h9m0.384534589s) [noise=true , address=45.91.116.11:4119 , piece_id=UWEB6A6U5QZT4FPVOSDLK2FADGE2ZLDUA6OZ6BU3ZX7DJCO22JTQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.195871761 (60h9m0.38617033s) [noise=true , address=45.91.116.12:4225 , piece_id=ZLHXC67OYXE4LHEOGPKYUDV6TMYVZNXXDN7LSBMGJHUEN5DIQXDA, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.195903069 (60h9m0.386201642s) [noise=true , address=45.91.116.10:4083 , piece_id=236G76NGG5I7QDZFZWSPHROVFVHU7VDO6G2QDVZSJKMVP2CLS75Q, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.205414322 (60h9m0.395712902s) [noise=true , address=45.91.116.8:4047 , piece_id=G4OOGIFPHZ7NYPJLRRILNUWRNHDJBISCCPH54VOJS2B7VSKRHY6Q, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.212962049 (60h9m0.403260632s) [noise=true , address=45.91.116.8:4048 , piece_id=VUMHPKQ3WBAY3VENB3IURNTWNGTNCTWITZGTGCDQD6B3A374JF6A, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.213778056 (60h9m0.404076629s) [noise=true , address=45.91.116.4:4937 , piece_id=IA6WH3XERNH243EI3MUD52NW5Y2SUKXOST75OXGKTJHUFQR65GQQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.218260079 (60h9m0.408558659s) [noise=true , address=45.91.116.6:4974 , piece_id=5JVTHPKIOZ6462BO7YGVQT2KATJTFCMJT327NJ5IGF5C44GTLPUQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:29.344821647 (60h9m0.53512022s) [noise=true , address=45.91.116.5:4298 , piece_id=KSQG47Y77JZOXKTEABFKFCCSYPLFX3AB5Q7Y7IZYPQ76YEDNN4WQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:30.609491777 (60h9m1.799790348s) [noise=true , address=77.90.31.13:4626 , piece_id=UJRAAHF5YIQQPN2BMAY2ATCWLPHLXGSOLOEWA43AFAOKV5TZBRSQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:30.625354579 (60h9m1.815653152s) [noise=true , address=77.90.31.13:4628 , piece_id=XXXBK35EZJQMBRGYOL2NQXWJT3YKHRSF5GBY5CWIXRK72TPUDO6A, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:48.490757774 (60h9m19.681056343s) [noise=true , address=104.36.82.14:4221 , piece_id=ED6BSVEMDN3MYEDJ45ZWEJQMI23OUI4ROW24CY2TDT4U6YJXEIWQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: <nil> Jul 26 23:44:58.036246465 (60h9m29.226545054s) [noise=true , address=104.36.82.13:4185 , piece_id=EIIBBWVVPXNFY3HYF564U6DRELZRNO3CIOBFK2D3PUHGVMGPWBSQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:37694->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:37694->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37694->104.36.82.13:4185: use of closed network connection Jul 26 23:44:58.036346671 (60h9m29.226645239s) [noise=true , address=104.36.82.13:4187 , piece_id=ESQ3FJOKTAVQS5ICEFYK5YX3OACI33MWLB2WB6V7UKHIUCMNUQAQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:56640->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:56640->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56640->104.36.82.13:4187: use of closed network connection Jul 26 23:44:58.036553964 (60h9m29.226852548s) [noise=true , address=104.36.82.2:4732 , piece_id=62UIZV5JB6UWLSWLN6JFK5P7VCJ4KB6VFUP6UZ4GCV2GNSL74EUQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 139.59.144.22:36846->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:36846->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36846->104.36.82.2:4732: use of closed network connection Jul 26 23:44:58.036709111 (60h9m29.227007692s) [noise=true , address=104.36.82.12:4149 , piece_id=2NNDBXJKQMJ7WIGMDESL5WYB3BLA5PZE6XKOSLYI53OC6NAXKWEQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 139.59.144.22:39004->104.36.82.12:4149: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.12:4149: use of closed network connection Jul 26 23:44:58.037024895 (60h9m29.22732348s) [noise=true , address=104.36.82.11:4115 , piece_id=OUDFQWGXQJOLL5YZ74TCZK3VO3VJDMPG55WKXXHBB7FRTD5PZHXQ, seg_pos={0 0}, upload=57673]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 139.59.144.22:54684->104.36.82.11:4115: use of closed network connection; write tcp 139.59.144.22:54684->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54684->104.36.82.11:4115: use of closed network connection Jul 26 23:44:58.037052826 (60h9m29.227351415s) [seg_pos={0 0}, upload=57673]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:44:58.147077685 (60h9m29.337376259s) [upload=57673]: Done uploading segments... Jul 26 23:44:58.147105738 (60h9m29.337404317s) [upload=57673]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.113132217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [X.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X.] duration: 44.639621542s FAIL (2025-07-26 23:44:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 386477011s) [upload=57629]: Done uploading segments... Jul 26 23:42:36.196198115 (60h7m7.386496688s) [upload=57629]: Upload finished. err: <nil> Jul 26 23:42:53.417482923 (60h7m24.607781508s) [upload=57646]: Starting upload Jul 26 23:42:53.417579840 (60h7m24.607878408s) [upload=57646]: Done starting upload Jul 26 23:42:53.417683172 (60h7m24.607981743s) [upload=57646]: Uploading segments... Jul 26 23:42:53.417899571 (60h7m24.608198145s) [upload=57646]: Got next segment. Inline: false Jul 26 23:42:53.417974540 (60h7m24.608273123s) [upload=57646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:53.417980468 (60h7m24.608279048s) [upload=57646]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:53.619456625 (60h7m24.809755192s) [seg_pos={0 0}, upload=57646]: Begin upload segment... Jul 26 23:42:53.619510776 (60h7m24.809809355s) [seg_pos={0 0}, upload=57646]: Done begin upload segment. Jul 26 23:42:53.619533396 (60h7m24.809831979s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620054445 (60h7m24.810353026s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620302982 (60h7m24.810601565s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620601932 (60h7m24.810900518s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621053086 (60h7m24.811351675s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621414282 (60h7m24.811712867s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621692545 (60h7m24.811991124s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622057938 (60h7m24.812356524s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622416760 (60h7m24.812715343s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622854791 (60h7m24.813155765s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623164302 (60h7m24.813462881s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623692679 (60h7m24.813991267s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624035038 (60h7m24.814333618s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624380099 (60h7m24.81467869s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624733702 (60h7m24.815032284s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625162941 (60h7m24.815461531s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625502379 (60h7m24.815800961s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625734416 (60h7m24.816032999s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:54.036062824 (60h7m25.226361413s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:54.190797058 (60h7m25.381095651s) [upload=57646]: Next returned nil segment Jul 26 23:42:54.190822148 (60h7m25.381120738s) [upload=57646]: Waiting for error group managing segments... Jul 26 23:42:54.375177273 (60h7m25.565475845s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.382307219 (60h7m25.572605792s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.908069114 (60h7m26.098367696s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.054218318 (60h7m26.244516891s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.134081796 (60h7m26.324380368s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:59.514001760 (60h7m30.70430033s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:00.521755754 (60h7m31.712054344s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:03.673252172 (60h7m34.863550755s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:13.898059806 (60h7m45.088358376s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:22.289157029 (60h7m53.479455602s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:37.181386006 (60h8m8.37168459s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:40.826146516 (60h8m12.016445106s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:42.389900272 (60h8m13.580198844s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:43.133590638 (60h8m14.323889205s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:53.139632762 (60h8m24.329931335s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139757243 (60h8m24.330055823s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139822221 (60h8m24.33012079s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: read tcp 139.59.144.22:38110->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:53.139855040 (60h8m24.330153626s) [seg_pos={0 0}, upload=57646]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:43:53.247153679 (60h8m24.437452249s) [upload=57646]: Done uploading segments... Jul 26 23:43:53.247179655 (60h8m24.437478234s) [upload=57646]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 44.639659165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X.......X........XX....X....X.X..X........XX.X....X......X] duration: 1m0.000501803s FAIL (2025-07-26 23:44:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000518649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [..X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XX] duration: 30.106570566s FAIL (2025-07-26 23:44:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.106590423s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [...X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..X] duration: 30.123520959s FAIL (2025-07-26 23:44:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.123547048s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX...] duration: 2.878780948s FAIL (2025-07-26 23:43:59 UTC) orbiter warning: context deadline exceeded Duration: 2.878825155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [....X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXX] duration: 30.112103408s FAIL (2025-07-26 23:43:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 0 0}, upload=57649]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:43:28.377742884 (60h7m59.568041454s) [upload=57649]: Done uploading segments... Jul 26 23:43:28.377772067 (60h7m59.568070647s) [upload=57649]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:43:28.804508040 (60h7m59.994806626s) [upload=57655]: Starting upload Jul 26 23:43:28.804618521 (60h7m59.9949171s) [upload=57655]: Done starting upload Jul 26 23:43:28.805028521 (60h7m59.995327091s) [upload=57655]: Uploading segments... Jul 26 23:43:28.805115791 (60h7m59.995414382s) [upload=57655]: Got next segment. Inline: false Jul 26 23:43:28.805162533 (60h7m59.99546112s) [upload=57655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:43:28.805168828 (60h7m59.995467415s) [upload=57655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:43:28.916136400 (60h8m0.106434987s) [seg_pos={0 0}, upload=57655]: Begin upload segment... Jul 26 23:43:28.916323297 (60h8m0.106621879s) [seg_pos={0 0}, upload=57655]: Done begin upload segment. Jul 26 23:43:28.916334451 (60h8m0.106633035s) [noise=true , address=45.91.116.10:4081 , piece_id=KZVH2PPEJOSAQHGVJ6LB7GY6YTTA6YRXS7QSTMLKWHO4YKQ6CLFA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.916379735 (60h8m0.106678325s) [noise=true , address=45.91.116.14:4189 , piece_id=5CFCVPJRIH3ZRGMPQOVRGOONISM5VPKHF4JL4EYIVCDI5GWX4Z6A, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.917390253 (60h8m0.107688842s) [noise=true , address=104.36.82.7:4912 , piece_id=U5PH52NJHCDKZF4OSVHRJS22CQ4WB7EY3EQEI2KZOP5RCTGLHGTA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.918088243 (60h8m0.108386831s) [noise=true , address=45.91.116.8:4045 , piece_id=6Q24RFXZ2FXJPSXQE2YFGSWOHUNGWSGG3NQFVEIZDD4K3NNOSXCA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.918195261 (60h8m0.108493879s) [noise=true , address=45.91.116.13:4153 , piece_id=5CBS7EP5HNNPRJIKI44AWQBU6AYZXH7S7HSEGIX32GPBFWOOHMIA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.918433438 (60h8m0.108732021s) [noise=true , address=104.36.82.13:4186 , piece_id=CPZ74HCAQZ36XGCN5HGWNDRREHHEJP4JB2ULCUIATS7FICRIBZKA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.918750532 (60h8m0.109049112s) [noise=true , address=45.91.116.5:4299 , piece_id=X6XKSHBWTIC4BPIWC6KLQ3TIRVEHKPYJ4QSB5F72WH44XZL4NTUA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.918990796 (60h8m0.10928939s) [noise=true , address=104.36.82.8:4971 , piece_id=NUGRIDGIE564MJK7R2CUXVD2QMUU73NV2L4NVQHWNLTDHCSEB7SQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.919071807 (60h8m0.109370377s) [noise=true , address=77.90.31.16:4735 , piece_id=FASFZ6UUPJIASNSURRZMGUEYXFNQMRJI3JYXBMAMFJL6PZN7FFRA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.919395666 (60h8m0.109694236s) [noise=true , address=45.91.116.11:4117 , piece_id=NSN5NATHVWDZ3O4N37HESOQG346KKJ75BSSXBK7XRGCVXXRNJRUQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.919696907 (60h8m0.109995486s) [noise=true , address=104.36.82.11:4114 , piece_id=A6P4CY6TRGNGG3WS4TP7S3FFJMYHWAA2RBI6M6QJP2DOD2Z7L4LQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.919746662 (60h8m0.110045282s) [noise=true , address=45.91.116.8:4047 , piece_id=OSX46KRZPG5OWIEDILUOJJBS5Z4XAZ5OMPZCZLK4WXOHCPOJLKKQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.920060333 (60h8m0.110358915s) [noise=true , address=45.91.116.4:4938 , piece_id=2DPUKWCFRBOO62YTYAOY3MEXW4LSLTAIYA3FPNMFKTOS2HWREKVQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.920429853 (60h8m0.110728435s) [noise=true , address=104.36.82.5:4837 , piece_id=ZV7SJ5D2QMDADUAKRJBUSYLWHRH2OBJQBGVPF2XONUS2DDEF3ZTA, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.920848164 (60h8m0.111146767s) [noise=true , address=104.36.82.3:4765 , piece_id=WDPGM7WY2CEJDROQFZOWVMODEYU5OIORXG7HISF4FBF7KQIF7MQQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:28.921188444 (60h8m0.111487055s) [noise=true , address=77.90.31.21:4878 , piece_id=EEVEVH42GFBYUW2BN3IYBVDKVBRI7TKJBI2B4PKDFKD4IY22ZRVQ, seg_pos={0 0}, upload=57655]: Uploading piece... Jul 26 23:43:29.524637277 (60h8m0.714935841s) [upload=57655]: Next returned nil segment Jul 26 23:43:29.524679367 (60h8m0.714977945s) [upload=57655]: Waiting for error group managing segments... Jul 26 23:43:29.667309441 (60h8m0.857608015s) [noise=true , address=45.91.116.8:4045 , piece_id=6Q24RFXZ2FXJPSXQE2YFGSWOHUNGWSGG3NQFVEIZDD4K3NNOSXCA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.667404834 (60h8m0.857703418s) [noise=true , address=45.91.116.11:4117 , piece_id=NSN5NATHVWDZ3O4N37HESOQG346KKJ75BSSXBK7XRGCVXXRNJRUQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.668333365 (60h8m0.858631931s) [noise=true , address=45.91.116.10:4081 , piece_id=KZVH2PPEJOSAQHGVJ6LB7GY6YTTA6YRXS7QSTMLKWHO4YKQ6CLFA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.677962940 (60h8m0.868261513s) [noise=true , address=45.91.116.13:4153 , piece_id=5CBS7EP5HNNPRJIKI44AWQBU6AYZXH7S7HSEGIX32GPBFWOOHMIA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.677987097 (60h8m0.868285678s) [noise=true , address=45.91.116.8:4047 , piece_id=OSX46KRZPG5OWIEDILUOJJBS5Z4XAZ5OMPZCZLK4WXOHCPOJLKKQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.679720641 (60h8m0.870019227s) [noise=true , address=45.91.116.4:4938 , piece_id=2DPUKWCFRBOO62YTYAOY3MEXW4LSLTAIYA3FPNMFKTOS2HWREKVQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:29.966141578 (60h8m1.156440151s) [noise=true , address=45.91.116.14:4189 , piece_id=5CFCVPJRIH3ZRGMPQOVRGOONISM5VPKHF4JL4EYIVCDI5GWX4Z6A, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:30.031355611 (60h8m1.221654176s) [noise=true , address=45.91.116.5:4299 , piece_id=X6XKSHBWTIC4BPIWC6KLQ3TIRVEHKPYJ4QSB5F72WH44XZL4NTUA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:51.195415961 (60h8m22.385714527s) [noise=true , address=104.36.82.13:4186 , piece_id=CPZ74HCAQZ36XGCN5HGWNDRREHHEJP4JB2ULCUIATS7FICRIBZKA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:56.072727506 (60h8m27.263026076s) [noise=true , address=104.36.82.5:4837 , piece_id=ZV7SJ5D2QMDADUAKRJBUSYLWHRH2OBJQBGVPF2XONUS2DDEF3ZTA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:56.253296109 (60h8m27.443594683s) [noise=true , address=77.90.31.21:4878 , piece_id=EEVEVH42GFBYUW2BN3IYBVDKVBRI7TKJBI2B4PKDFKD4IY22ZRVQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: <nil> Jul 26 23:43:58.516656855 (60h8m29.706955435s) [noise=true , address=104.36.82.8:4971 , piece_id=NUGRIDGIE564MJK7R2CUXVD2QMUU73NV2L4NVQHWNLTDHCSEB7SQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:59150->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:59150->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->104.36.82.8:4971: use of closed network connection Jul 26 23:43:58.516734043 (60h8m29.707032623s) [noise=true , address=104.36.82.7:4912 , piece_id=U5PH52NJHCDKZF4OSVHRJS22CQ4WB7EY3EQEI2KZOP5RCTGLHGTA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 139.59.144.22:47126->104.36.82.7:4912: use of closed network connection; write tcp 139.59.144.22:47126->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47126->104.36.82.7:4912: use of closed network connection Jul 26 23:43:58.516815155 (60h8m29.707113724s) [noise=true , address=104.36.82.11:4114 , piece_id=A6P4CY6TRGNGG3WS4TP7S3FFJMYHWAA2RBI6M6QJP2DOD2Z7L4LQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:58.517021935 (60h8m29.707320514s) [noise=true , address=104.36.82.3:4765 , piece_id=WDPGM7WY2CEJDROQFZOWVMODEYU5OIORXG7HISF4FBF7KQIF7MQQ, seg_pos={0 0}, upload=57655]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 139.59.144.22:50142->104.36.82.3:4765: use of closed network connection; write tcp 139.59.144.22:50142->104.36.82.3:4765: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50142->104.36.82.3:4765: use of closed network connection Jul 26 23:43:58.517045328 (60h8m29.707343895s) [noise=true , address=77.90.31.16:4735 , piece_id=FASFZ6UUPJIASNSURRZMGUEYXFNQMRJI3JYXBMAMFJL6PZN7FFRA, seg_pos={0 0}, upload=57655]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:38962->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:58.517133598 (60h8m29.707432187s) [seg_pos={0 0}, upload=57655]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:43:58.626722407 (60h8m29.817020994s) [upload=57655]: Done uploading segments... Jul 26 23:43:58.626754315 (60h8m29.817052902s) [upload=57655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.112122882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X] duration: 1m0.109648049s FAIL (2025-07-26 23:43:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 386477011s) [upload=57629]: Done uploading segments... Jul 26 23:42:36.196198115 (60h7m7.386496688s) [upload=57629]: Upload finished. err: <nil> Jul 26 23:42:53.417482923 (60h7m24.607781508s) [upload=57646]: Starting upload Jul 26 23:42:53.417579840 (60h7m24.607878408s) [upload=57646]: Done starting upload Jul 26 23:42:53.417683172 (60h7m24.607981743s) [upload=57646]: Uploading segments... Jul 26 23:42:53.417899571 (60h7m24.608198145s) [upload=57646]: Got next segment. Inline: false Jul 26 23:42:53.417974540 (60h7m24.608273123s) [upload=57646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:53.417980468 (60h7m24.608279048s) [upload=57646]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:53.619456625 (60h7m24.809755192s) [seg_pos={0 0}, upload=57646]: Begin upload segment... Jul 26 23:42:53.619510776 (60h7m24.809809355s) [seg_pos={0 0}, upload=57646]: Done begin upload segment. Jul 26 23:42:53.619533396 (60h7m24.809831979s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620054445 (60h7m24.810353026s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620302982 (60h7m24.810601565s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.620601932 (60h7m24.810900518s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621053086 (60h7m24.811351675s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621414282 (60h7m24.811712867s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.621692545 (60h7m24.811991124s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622057938 (60h7m24.812356524s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622416760 (60h7m24.812715343s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.622854791 (60h7m24.813155765s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623164302 (60h7m24.813462881s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.623692679 (60h7m24.813991267s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624035038 (60h7m24.814333618s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624380099 (60h7m24.81467869s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.624733702 (60h7m24.815032284s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625162941 (60h7m24.815461531s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625502379 (60h7m24.815800961s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:53.625734416 (60h7m24.816032999s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Uploading piece... Jul 26 23:42:54.036062824 (60h7m25.226361413s) [noise=true , address=79.127.219.51:4210 , piece_id=ZZUGETUNAHDAORZ5IUSE5R7IFVE4ZBALGESZHMDUJRBT4Z4EQENA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:42:54.190797058 (60h7m25.381095651s) [upload=57646]: Next returned nil segment Jul 26 23:42:54.190822148 (60h7m25.381120738s) [upload=57646]: Waiting for error group managing segments... Jul 26 23:42:54.375177273 (60h7m25.565475845s) [noise=true , address=79.127.219.38:4005 , piece_id=TGSDOITTHPXVVYDXOQ2ZLYAYK5IHU55L2Y3RAYFO7I47ILSZOVYA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.382307219 (60h7m25.572605792s) [noise=true , address=77.90.31.11:4560 , piece_id=TRNIME5RZLLAMMRSPRAJEO5R3B2P2CLQRNB7RYAZKZOMHFE7W7LQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:54.908069114 (60h7m26.098367696s) [noise=true , address=64.49.13.140:4508 , piece_id=L2LS34LQKDTR2GTEAX66XEOOJAA3LRKCAWM2OWDRXGMK44NFAQQQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.054218318 (60h7m26.244516891s) [noise=true , address=79.127.205.245:4251 , piece_id=ZLZGGK46SW4QFCQEL2IBZCRYG72GLEQQ2ENK25FDCISCBGTFNLJQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:55.134081796 (60h7m26.324380368s) [noise=true , address=79.127.205.235:4686 , piece_id=TGQ2MEWLFFDBWRHOKXHPRBJJQCRAEYT5YEDRXXST5M7GZKUMK6AA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:42:59.514001760 (60h7m30.70430033s) [noise=true , address=156.146.43.225:4434 , piece_id=UXBCHBNMSJQXKDT3FRWB4G5TTNS3ZIRTRWYSB6AFZFCLWAY3ONHQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:00.521755754 (60h7m31.712054344s) [noise=true , address=45.91.116.9:4267 , piece_id=6TBYYFP3TDYCD6GNRCXMRGR52SNENNXLVUIH55QYWALQVKDGB3NA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:43:03.673252172 (60h7m34.863550755s) [noise=true , address=79.127.220.98:4421 , piece_id=U3K5OEUAD3DQXTN2D54CA7NSLPLLK3KOF7V7B5GX2J2HZNCX3TAA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:13.898059806 (60h7m45.088358376s) [noise=true , address=104.36.82.54:4185 , piece_id=CYQLGCJYAEFQHBV7WBUFMIQO4ZZSEUP72BXHAZDEQD5FTZ4YFGWA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:22.289157029 (60h7m53.479455602s) [noise=true , address=104.36.82.53:4137 , piece_id=WTBSL7LAOK3GTKSIEW4KJQIUTYETDMJ75JW56CWTVHZVZ3XLP2FQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:37.181386006 (60h8m8.37168459s) [noise=true , address=79.127.201.215:4350 , piece_id=R7JTVOVK7SU77UPONG2RYKLEUPDLJ3MFB2B4J5T4DM5SN3J4Y33A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:40.826146516 (60h8m12.016445106s) [noise=true , address=104.36.82.56:4257 , piece_id=COG4WS5OOHEBAXAEPGAQJ3VAYVZHXUWVW3NNKVY7SOSQEMVNZB6A, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:42.389900272 (60h8m13.580198844s) [noise=true , address=104.36.82.47:4519 , piece_id=OOUS42DJIX2LVFRGWSI2AKSSA6GOXZR2UCKZN4RPZPDMNCM7FLLA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:43.133590638 (60h8m14.323889205s) [noise=true , address=104.36.82.46:4501 , piece_id=2E2GGCDDHQQJYFMRIUASCAGO7ZC47TUZFROP6CU73GCQAULI3FPA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: <nil> Jul 26 23:43:53.139632762 (60h8m24.329931335s) [noise=true , address=77.90.31.15:4720 , piece_id=7HCOCS5IBMMDDCUNHOX5L5A2ABHNZTSZRJRZQIRG7A3OYWGNZBOA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139757243 (60h8m24.330055823s) [noise=true , address=104.36.82.9:4022 , piece_id=JGHV7GS2O5HMJJDDAOGXHOFR443YK3GKQNXHIAJBESC7HGWF3VEA, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:43:53.139822221 (60h8m24.33012079s) [noise=true , address=77.90.31.10:4528 , piece_id=UURUBV336XU7ILWKZSMYKARQZTI6LBOXPU5TY3MCXB4HNDOTPNZQ, seg_pos={0 0}, upload=57646]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: read tcp 139.59.144.22:38110->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:53.139855040 (60h8m24.330153626s) [seg_pos={0 0}, upload=57646]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:43:53.247153679 (60h8m24.437452249s) [upload=57646]: Done uploading segments... Jul 26 23:43:53.247179655 (60h8m24.437478234s) [upload=57646]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.109674288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X........XX....X....X.X..X........XX.X....X......] duration: 37.543044931s FAIL (2025-07-26 23:43:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.543065783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX..] duration: 2.997916774s FAIL (2025-07-26 23:43:39 UTC) orbiter warning: context deadline exceeded Duration: 2.998451148s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [....X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..] duration: 29.816714345s FAIL (2025-07-26 23:43:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 29.816746121s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [.....X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXX] duration: 30.110075437s FAIL (2025-07-26 23:43:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ent. successful: 9 optimal: 13 errs: <nil> Jul 26 23:42:39.955734530 (60h7m11.1460331s) [upload=57633]: Done uploading segments... Jul 26 23:42:39.955763362 (60h7m11.146061944s) [upload=57633]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:42:58.547796721 (60h7m29.738095312s) [upload=57649]: Starting upload Jul 26 23:42:58.547885533 (60h7m29.738184121s) [upload=57649]: Done starting upload Jul 26 23:42:58.548710922 (60h7m29.739009505s) [upload=57649]: Uploading segments... Jul 26 23:42:58.548786432 (60h7m29.73908502s) [upload=57649]: Got next segment. Inline: false Jul 26 23:42:58.548924850 (60h7m29.739223443s) [upload=57649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:58.548934344 (60h7m29.739232928s) [upload=57649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:58.796171659 (60h7m29.986470243s) [seg_pos={0 0}, upload=57649]: Begin upload segment... Jul 26 23:42:58.796232441 (60h7m29.986531025s) [seg_pos={0 0}, upload=57649]: Done begin upload segment. Jul 26 23:42:58.796264595 (60h7m29.986563184s) [noise=true , address=45.91.116.8:4048 , piece_id=GDCQNXOIBL4XCU6FXXXSOXZCAUCB6DYUVBTNBMOXQARAJIQ2BFTA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.796318675 (60h7m29.986617261s) [noise=true , address=104.36.82.16:4294 , piece_id=DZWHHXTTAMXI2DM5QPFJ57JQOHNUQ6JHLYDZVPZEOTW7UTPDGWFA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.796712250 (60h7m29.987010832s) [noise=true , address=104.36.82.6:4873 , piece_id=74LNPF3SKIBMBW2GUDO3T5BL65NT7GYFQXGVDDJSXWKBX4SX3Q5A, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797068400 (60h7m29.987366988s) [noise=true , address=104.36.82.12:4151 , piece_id=JIQLKZ3YZ3XQDJ3BG6JEACNGT72OOJJ2COHXS2YAT3EM3MVQ7HDA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797232583 (60h7m29.987531165s) [noise=true , address=104.36.82.6:4874 , piece_id=JLJ3JNFH7DPCPVJWLPYLXZWJYM3VMQGAF3WSWFRYHTECIELDFA4Q, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797486373 (60h7m29.987784952s) [noise=true , address=104.36.82.3:4765 , piece_id=QNDMDWVXG4ICQAL2VV4WIAMR6OEW7PBI7VGD3XJAZM3URMBI5MNA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797529809 (60h7m29.987828404s) [noise=true , address=104.36.82.7:4911 , piece_id=5JXUZHQ6LMIHKWDHA5557HKNQOGDJDI2BNH65NNIAHYUAVHKDGDA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797818953 (60h7m29.988117536s) [noise=true , address=104.36.82.13:4185 , piece_id=LAKOHX2RL5PPAGRW7UXCKWGBOAEQM5Y6YVM662Q7LSXHM67CCCRQ, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.797876010 (60h7m29.988174592s) [noise=true , address=77.90.31.21:4879 , piece_id=VU36GG2IAETEJHW7JNXJMDALXR4LW2UO5FNFTYGPOVU54N5IW6EA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.798241735 (60h7m29.98854032s) [noise=true , address=77.90.31.16:4734 , piece_id=FDPZEO7FA5JHDE7SI4IPYXUXFNIVX4CYPMQ6NFIQX53Y76QBKTIQ, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.798336118 (60h7m29.988634701s) [noise=true , address=104.36.82.2:4731 , piece_id=WMYQQ4L22J35BUZ4PRCR6VUW7LMQ2P66QWIJUC32TUYPFX33INXA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.798605124 (60h7m29.988903762s) [noise=true , address=104.36.82.8:4969 , piece_id=UG7R7H2AAZAYSGTU2GCDYH4BEMR4JN7CSNSL7BS25POK25PHYVXA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.798739961 (60h7m29.989038546s) [noise=true , address=45.91.116.8:4045 , piece_id=CJ45MXEWGQNOGO6C27VR74SGDFZNTNJTJOINO6VX6QWHMTCSYRCA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.798887606 (60h7m29.989186191s) [noise=true , address=77.90.31.15:4697 , piece_id=H6DGX5TQAS44UOEJ34PWNC4ZGZRT2XJASI636FHKNX3NGBZV52BA, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.799136744 (60h7m29.989435327s) [noise=true , address=104.36.82.10:4078 , piece_id=JLD4XA3DCCKX4WB72UOXJENJVMTORPPD26E7OVT34ZVOAAM46G3Q, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:58.799195573 (60h7m29.989494161s) [noise=true , address=104.36.82.5:4838 , piece_id=FEOL4U6B5LBLDJIXPZAFEX6KHWEPRL2AQ3QTSNZLEIEB32B5TS4A, seg_pos={0 0}, upload=57649]: Uploading piece... Jul 26 23:42:59.405505885 (60h7m30.595804469s) [upload=57649]: Next returned nil segment Jul 26 23:42:59.405530973 (60h7m30.59582954s) [upload=57649]: Waiting for error group managing segments... Jul 26 23:42:59.542228999 (60h7m30.732527568s) [noise=true , address=45.91.116.8:4045 , piece_id=CJ45MXEWGQNOGO6C27VR74SGDFZNTNJTJOINO6VX6QWHMTCSYRCA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:42:59.543957244 (60h7m30.734255834s) [noise=true , address=45.91.116.8:4048 , piece_id=GDCQNXOIBL4XCU6FXXXSOXZCAUCB6DYUVBTNBMOXQARAJIQ2BFTA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:09.130882285 (60h7m40.321180865s) [noise=true , address=77.90.31.15:4697 , piece_id=H6DGX5TQAS44UOEJ34PWNC4ZGZRT2XJASI636FHKNX3NGBZV52BA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:09.938268582 (60h7m41.128567162s) [noise=true , address=104.36.82.5:4838 , piece_id=FEOL4U6B5LBLDJIXPZAFEX6KHWEPRL2AQ3QTSNZLEIEB32B5TS4A, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:13.462159508 (60h7m44.652458087s) [noise=true , address=104.36.82.10:4078 , piece_id=JLD4XA3DCCKX4WB72UOXJENJVMTORPPD26E7OVT34ZVOAAM46G3Q, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:16.379731795 (60h7m47.570030361s) [noise=true , address=104.36.82.7:4911 , piece_id=5JXUZHQ6LMIHKWDHA5557HKNQOGDJDI2BNH65NNIAHYUAVHKDGDA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:17.768641521 (60h7m48.958940098s) [noise=true , address=104.36.82.12:4151 , piece_id=JIQLKZ3YZ3XQDJ3BG6JEACNGT72OOJJ2COHXS2YAT3EM3MVQ7HDA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:19.324667611 (60h7m50.514966176s) [noise=true , address=104.36.82.3:4765 , piece_id=QNDMDWVXG4ICQAL2VV4WIAMR6OEW7PBI7VGD3XJAZM3URMBI5MNA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:21.116414538 (60h7m52.306713107s) [noise=true , address=104.36.82.13:4185 , piece_id=LAKOHX2RL5PPAGRW7UXCKWGBOAEQM5Y6YVM662Q7LSXHM67CCCRQ, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:22.382776505 (60h7m53.57307508s) [noise=true , address=104.36.82.16:4294 , piece_id=DZWHHXTTAMXI2DM5QPFJ57JQOHNUQ6JHLYDZVPZEOTW7UTPDGWFA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:23.607241526 (60h7m54.79754011s) [noise=true , address=104.36.82.2:4731 , piece_id=WMYQQ4L22J35BUZ4PRCR6VUW7LMQ2P66QWIJUC32TUYPFX33INXA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: <nil> Jul 26 23:43:28.269796065 (60h7m59.460094644s) [noise=true , address=104.36.82.6:4874 , piece_id=JLJ3JNFH7DPCPVJWLPYLXZWJYM3VMQGAF3WSWFRYHTECIELDFA4Q, seg_pos={0 0}, upload=57649]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:47962->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:47962->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47962->104.36.82.6:4874: use of closed network connection Jul 26 23:43:28.269852405 (60h7m59.460150984s) [noise=true , address=104.36.82.8:4969 , piece_id=UG7R7H2AAZAYSGTU2GCDYH4BEMR4JN7CSNSL7BS25POK25PHYVXA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:48228->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:48228->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48228->104.36.82.8:4969: use of closed network connection Jul 26 23:43:28.269975803 (60h7m59.460274384s) [noise=true , address=77.90.31.16:4734 , piece_id=FDPZEO7FA5JHDE7SI4IPYXUXFNIVX4CYPMQ6NFIQX53Y76QBKTIQ, seg_pos={0 0}, upload=57649]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:28.270136748 (60h7m59.460435327s) [noise=true , address=104.36.82.6:4873 , piece_id=74LNPF3SKIBMBW2GUDO3T5BL65NT7GYFQXGVDDJSXWKBX4SX3Q5A, seg_pos={0 0}, upload=57649]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 139.59.144.22:56586->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:56586->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56586->104.36.82.6:4873: use of closed network connection Jul 26 23:43:28.270301834 (60h7m59.460600413s) [noise=true , address=77.90.31.21:4879 , piece_id=VU36GG2IAETEJHW7JNXJMDALXR4LW2UO5FNFTYGPOVU54N5IW6EA, seg_pos={0 0}, upload=57649]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 139.59.144.22:43452->77.90.31.21:4879: use of closed network connection; context deadline exceeded Jul 26 23:43:28.270319594 (60h7m59.460618164s) [seg_pos={0 0}, upload=57649]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:43:28.377742884 (60h7m59.568041454s) [upload=57649]: Done uploading segments... Jul 26 23:43:28.377772067 (60h7m59.568070647s) [upload=57649]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.110116862s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX.] duration: 20.328774934s FAIL (2025-07-26 23:43:04 UTC) orbiter warning: context deadline exceeded Duration: 20.328799807s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [......X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXX] duration: 30.104964498s FAIL (2025-07-26 23:42:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== , piece_id=LQRJMHZ2ZWHTCAQRJFNTQUHAYPN6XZZTR5FFSMHTP6DUTFXTP2IQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.236500560 (60h6m41.426799132s) [noise=true , address=104.36.82.11:4113 , piece_id=5NDPYML5IVXMRHGN25M5YEZ7RVUWEX3SG2RDD5HJYUYGFYNGURXA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.237116549 (60h6m41.427415126s) [noise=true , address=45.91.116.7:4009 , piece_id=X7CNXYYA64KKU5VU26DZAXFNX7RQR7B6DVVC5UVFRD655XHPIDXQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.238518180 (60h6m41.428816752s) [noise=true , address=45.91.116.9:4261 , piece_id=X776FMH2PYY5E6XLCQPBL7QQJCMNBE365YIDK7DFGB473EE5CEPA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.238850711 (60h6m41.429149277s) [noise=true , address=45.91.116.13:4155 , piece_id=BRYTD77BRVDZJTOB7U227KNT6XKSZOZ5OWCLTK2HONBPUQCX2GYA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.239348942 (60h6m41.429647515s) [noise=true , address=104.36.82.6:4875 , piece_id=L3URQNANIYFOJXECGRYRIH6H2BRYZMUVJ52XE6PVFGDNNGMAEEAA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.239819979 (60h6m41.430118545s) [noise=true , address=45.91.116.4:4940 , piece_id=ETTQFEBMDT4JJDQQJKK4WCCXT6IPGJ6F7OGGZ3VVVZTIF4N3AMFQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.240192441 (60h6m41.430491006s) [noise=true , address=45.91.116.12:4225 , piece_id=OOTKEF67WXZBQEJ36FJXJE6FMJ2APIUSFF775GNCMF6SSQ3FV54Q, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.241379767 (60h6m41.43167834s) [noise=true , address=77.90.31.12:4589 , piece_id=QV7AF6NCS3P42K473S4BW6MY553FCFJ4DLICZ6WTIDVUX65KHCMQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.241867356 (60h6m41.432165929s) [noise=true , address=104.36.82.14:4223 , piece_id=TIYAMF4EWDXMF7QG5DHT4QV7WAF6LUX5YC4MKAZRNQLTAY6EVHJQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.242242105 (60h6m41.432540682s) [noise=true , address=45.91.116.7:4010 , piece_id=5JP5B43ZVASLSGJYA75276OIIU4G54HYPXWN2WEXB3XVANF3SODA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.242369634 (60h6m41.432668203s) [noise=true , address=104.36.82.10:4079 , piece_id=LQTTXLQFNDSTEDHIKPPXDZ66S2ZJHTD3NGMJDZYOBGTMNOWYTOVA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.242718270 (60h6m41.433016841s) [noise=true , address=104.36.82.16:4294 , piece_id=7OTNV2KSYM6HEQTVFBHOSSLYIJTFJCW2J3IE7RNF3SYY42XUKGQQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.242856968 (60h6m41.433155539s) [noise=true , address=77.90.31.18:4805 , piece_id=TPPXBBWKZYNGEMLWSNHMHRPHJ227UKCPW7BXGZDRQGQ4WPXKOMFQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.243217539 (60h6m41.433516111s) [noise=true , address=104.36.82.13:4185 , piece_id=ZYVNEALOLA5A26CWEFRTAI227SSKAVRO2EZGQHOLUILHVX5AZ3QQ, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.243330988 (60h6m41.433629555s) [noise=true , address=104.36.82.5:4837 , piece_id=GUIBFEFSLIZMT7N7JV2SXWOT7FQFLACUAOXMI3QT4HDWESYLVDBA, seg_pos={0 0}, upload=57633]: Uploading piece... Jul 26 23:42:10.853407837 (60h6m42.043706405s) [upload=57633]: Next returned nil segment Jul 26 23:42:10.853424478 (60h6m42.043723061s) [upload=57633]: Waiting for error group managing segments... Jul 26 23:42:10.985801086 (60h6m42.176099657s) [noise=true , address=45.91.116.8:4047 , piece_id=LQRJMHZ2ZWHTCAQRJFNTQUHAYPN6XZZTR5FFSMHTP6DUTFXTP2IQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:10.987924388 (60h6m42.178222971s) [noise=true , address=45.91.116.13:4155 , piece_id=BRYTD77BRVDZJTOB7U227KNT6XKSZOZ5OWCLTK2HONBPUQCX2GYA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:10.988991057 (60h6m42.179289648s) [noise=true , address=45.91.116.4:4940 , piece_id=ETTQFEBMDT4JJDQQJKK4WCCXT6IPGJ6F7OGGZ3VVVZTIF4N3AMFQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:10.990269805 (60h6m42.18056837s) [noise=true , address=45.91.116.12:4225 , piece_id=OOTKEF67WXZBQEJ36FJXJE6FMJ2APIUSFF775GNCMF6SSQ3FV54Q, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:10.999504110 (60h6m42.189802697s) [noise=true , address=45.91.116.7:4009 , piece_id=X7CNXYYA64KKU5VU26DZAXFNX7RQR7B6DVVC5UVFRD655XHPIDXQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:11.006145136 (60h6m42.196443726s) [noise=true , address=45.91.116.7:4010 , piece_id=5JP5B43ZVASLSGJYA75276OIIU4G54HYPXWN2WEXB3XVANF3SODA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:11.010617071 (60h6m42.200915658s) [noise=true , address=77.90.31.12:4589 , piece_id=QV7AF6NCS3P42K473S4BW6MY553FCFJ4DLICZ6WTIDVUX65KHCMQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:11.209121355 (60h6m42.399419924s) [noise=true , address=45.91.116.9:4261 , piece_id=X776FMH2PYY5E6XLCQPBL7QQJCMNBE365YIDK7DFGB473EE5CEPA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:12.053765029 (60h6m43.244063619s) [noise=true , address=77.90.31.18:4805 , piece_id=TPPXBBWKZYNGEMLWSNHMHRPHJ227UKCPW7BXGZDRQGQ4WPXKOMFQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: <nil> Jul 26 23:42:39.852387978 (60h7m11.042686545s) [noise=true , address=104.36.82.13:4185 , piece_id=ZYVNEALOLA5A26CWEFRTAI227SSKAVRO2EZGQHOLUILHVX5AZ3QQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:43418->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:43418->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43418->104.36.82.13:4185: use of closed network connection Jul 26 23:42:39.852445221 (60h7m11.042743802s) [noise=true , address=104.36.82.5:4837 , piece_id=GUIBFEFSLIZMT7N7JV2SXWOT7FQFLACUAOXMI3QT4HDWESYLVDBA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:52958->104.36.82.5:4837: use of closed network connection; write tcp 139.59.144.22:52958->104.36.82.5:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52958->104.36.82.5:4837: use of closed network connection Jul 26 23:42:39.852482426 (60h7m11.042780996s) [noise=true , address=104.36.82.14:4223 , piece_id=TIYAMF4EWDXMF7QG5DHT4QV7WAF6LUX5YC4MKAZRNQLTAY6EVHJQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:53068->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:53068->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53068->104.36.82.14:4223: use of closed network connection Jul 26 23:42:39.852527388 (60h7m11.042825958s) [noise=true , address=104.36.82.6:4875 , piece_id=L3URQNANIYFOJXECGRYRIH6H2BRYZMUVJ52XE6PVFGDNNGMAEEAA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 139.59.144.22:40582->104.36.82.6:4875: use of closed network connection; write tcp 139.59.144.22:40582->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40582->104.36.82.6:4875: use of closed network connection Jul 26 23:42:39.852555043 (60h7m11.042853618s) [noise=true , address=104.36.82.16:4294 , piece_id=7OTNV2KSYM6HEQTVFBHOSSLYIJTFJCW2J3IE7RNF3SYY42XUKGQQ, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:47440->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:47440->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47440->104.36.82.16:4294: use of closed network connection Jul 26 23:42:39.852593294 (60h7m11.042891862s) [noise=true , address=104.36.82.10:4079 , piece_id=LQTTXLQFNDSTEDHIKPPXDZ66S2ZJHTD3NGMJDZYOBGTMNOWYTOVA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47906->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:47906->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47906->104.36.82.10:4079: use of closed network connection Jul 26 23:42:39.852634503 (60h7m11.042933071s) [noise=true , address=104.36.82.11:4113 , piece_id=5NDPYML5IVXMRHGN25M5YEZ7RVUWEX3SG2RDD5HJYUYGFYNGURXA, seg_pos={0 0}, upload=57633]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:41830->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:41830->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41830->104.36.82.11:4113: use of closed network connection Jul 26 23:42:39.852667997 (60h7m11.042966606s) [seg_pos={0 0}, upload=57633]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:42:39.955734530 (60h7m11.1460331s) [upload=57633]: Done uploading segments... Jul 26 23:42:39.955763362 (60h7m11.146061944s) [upload=57633]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.104991326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..] duration: 56.159009187s FAIL (2025-07-26 23:42:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== pload=57600]: Got next segment. Inline: false Jul 26 23:39:59.640133629 (60h4m30.83043221s) [upload=57600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:59.640172130 (60h4m30.830470709s) [upload=57600]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:59.836029504 (60h4m31.026328083s) [seg_pos={0 0}, upload=57600]: Begin upload segment... Jul 26 23:39:59.836088725 (60h4m31.026387294s) [seg_pos={0 0}, upload=57600]: Done begin upload segment. Jul 26 23:39:59.836122384 (60h4m31.026420962s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836155585 (60h4m31.026454174s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836522794 (60h4m31.026821381s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.837656627 (60h4m31.027955209s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838186932 (60h4m31.028485513s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838261347 (60h4m31.02855993s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838974935 (60h4m31.029273521s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839024475 (60h4m31.029323056s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839824134 (60h4m31.030122716s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839893733 (60h4m31.030192318s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.840657395 (60h4m31.030955998s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841380506 (60h4m31.031679096s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841775028 (60h4m31.03207361s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843059453 (60h4m31.033358038s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843511089 (60h4m31.033809675s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.844586849 (60h4m31.034885436s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845046976 (60h4m31.035345569s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845834257 (60h4m31.036132846s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:40:00.366525002 (60h4m31.556823573s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:00.378498951 (60h4m31.568797521s) [upload=57600]: Next returned nil segment Jul 26 23:40:00.378899577 (60h4m31.569198163s) [upload=57600]: Waiting for error group managing segments... Jul 26 23:40:00.573447807 (60h4m31.763746387s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.593073616 (60h4m31.783372184s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.634561036 (60h4m31.824859609s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.654282381 (60h4m31.844580956s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.144531131 (60h4m32.334829714s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.201706894 (60h4m32.39200549s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.372568543 (60h4m32.562867135s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:22.567473269 (60h4m53.757771848s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:30.015266116 (60h5m1.205564685s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:32.139831734 (60h5m3.330130317s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:41.195176201 (60h5m12.385474787s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:46.287007839 (60h5m17.477306424s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:48.854794351 (60h5m20.04509292s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:59.366831107 (60h5m30.55712968s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:129av1sWniou1aWC1EReoKyaxztX2g6bQwzSFpGqwxx3ZQ3nz5e, address:104.36.82.8:4976): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367098403 (60h5m30.557396979s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:104.36.82.6:4890): protocol: expected piece hash; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; EOF Jul 26 23:40:59.367280703 (60h5m30.557579272s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection Jul 26 23:40:59.367402645 (60h5m30.557701214s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367432324 (60h5m30.557730904s) [seg_pos={0 0}, upload=57600]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:40:59.483361686 (60h5m30.673660258s) [upload=57600]: Done uploading segments... Jul 26 23:40:59.483390751 (60h5m30.67368933s) [upload=57600]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 56.159059347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....X.......X........XX....X....X.X..X........XX.X....X.....] duration: 41.874538458s FAIL (2025-07-26 23:42:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.874566978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [.....X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX.] duration: 3.974454934s FAIL (2025-07-26 23:42:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 3.974486672s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [......X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX] duration: 30.108615496s FAIL (2025-07-26 23:42:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.108631116s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XXX] duration: 30.001106344s FAIL (2025-07-26 23:42:07 UTC) orbiter warning: context deadline exceeded Duration: 30.001140492s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [.......X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXX] duration: 30.140091559s FAIL (2025-07-26 23:41:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.140121507s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X.] duration: 43.970656688s FAIL (2025-07-26 23:41:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== pload=57600]: Got next segment. Inline: false Jul 26 23:39:59.640133629 (60h4m30.83043221s) [upload=57600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:59.640172130 (60h4m30.830470709s) [upload=57600]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:59.836029504 (60h4m31.026328083s) [seg_pos={0 0}, upload=57600]: Begin upload segment... Jul 26 23:39:59.836088725 (60h4m31.026387294s) [seg_pos={0 0}, upload=57600]: Done begin upload segment. Jul 26 23:39:59.836122384 (60h4m31.026420962s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836155585 (60h4m31.026454174s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836522794 (60h4m31.026821381s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.837656627 (60h4m31.027955209s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838186932 (60h4m31.028485513s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838261347 (60h4m31.02855993s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838974935 (60h4m31.029273521s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839024475 (60h4m31.029323056s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839824134 (60h4m31.030122716s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839893733 (60h4m31.030192318s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.840657395 (60h4m31.030955998s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841380506 (60h4m31.031679096s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841775028 (60h4m31.03207361s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843059453 (60h4m31.033358038s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843511089 (60h4m31.033809675s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.844586849 (60h4m31.034885436s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845046976 (60h4m31.035345569s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845834257 (60h4m31.036132846s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:40:00.366525002 (60h4m31.556823573s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:00.378498951 (60h4m31.568797521s) [upload=57600]: Next returned nil segment Jul 26 23:40:00.378899577 (60h4m31.569198163s) [upload=57600]: Waiting for error group managing segments... Jul 26 23:40:00.573447807 (60h4m31.763746387s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.593073616 (60h4m31.783372184s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.634561036 (60h4m31.824859609s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.654282381 (60h4m31.844580956s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.144531131 (60h4m32.334829714s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.201706894 (60h4m32.39200549s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.372568543 (60h4m32.562867135s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:22.567473269 (60h4m53.757771848s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:30.015266116 (60h5m1.205564685s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:32.139831734 (60h5m3.330130317s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:41.195176201 (60h5m12.385474787s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:46.287007839 (60h5m17.477306424s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:48.854794351 (60h5m20.04509292s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:59.366831107 (60h5m30.55712968s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:129av1sWniou1aWC1EReoKyaxztX2g6bQwzSFpGqwxx3ZQ3nz5e, address:104.36.82.8:4976): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367098403 (60h5m30.557396979s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:104.36.82.6:4890): protocol: expected piece hash; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; EOF Jul 26 23:40:59.367280703 (60h5m30.557579272s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection Jul 26 23:40:59.367402645 (60h5m30.557701214s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367432324 (60h5m30.557730904s) [seg_pos={0 0}, upload=57600]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:40:59.483361686 (60h5m30.673660258s) [upload=57600]: Done uploading segments... Jul 26 23:40:59.483390751 (60h5m30.67368933s) [upload=57600]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 43.970683557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.......X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXX] duration: 30.110346528s FAIL (2025-07-26 23:41:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 26 23:41:04.097888586 (60h5m35.288187155s) [upload=57618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:41:04.206163590 (60h5m35.396462159s) [seg_pos={0 0}, upload=57618]: Begin upload segment... Jul 26 23:41:04.206237689 (60h5m35.396536258s) [seg_pos={0 0}, upload=57618]: Done begin upload segment. Jul 26 23:41:04.206385865 (60h5m35.396684456s) [noise=true , address=104.36.82.11:4113 , piece_id=5YLHKA2L5F75OE56XTFGDNU4BHONFFLWT2ICI7EYOOLD7LQORFYQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.206449587 (60h5m35.396748169s) [noise=true , address=77.90.31.14:4662 , piece_id=L23MCSL2BLZRHR3M3ABEJ3GBL4LOVYASILFDNKKLGSOD2WCQEXIQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.206829502 (60h5m35.397128081s) [noise=true , address=77.90.31.13:4626 , piece_id=DB4H2EWGUP37SVU5EGM44O5C5AGFJ4GQJM4DMOLIE5ZPNMYNYVZQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.207881715 (60h5m35.398180296s) [noise=true , address=77.90.31.18:4807 , piece_id=WOMXZGRHNTCUIWN2GMOVLCUPK4FHMEV72RNKRZU4ACGA25ZUEF5A, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.208453196 (60h5m35.398751767s) [noise=true , address=77.90.31.19:4843 , piece_id=33NPNN5P57MT2PNG5MPYURE6NCOZ43ZF5WF2YHBAZ3577YHSANSQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.208991751 (60h5m35.399290332s) [noise=true , address=45.91.116.13:4154 , piece_id=IHBPYX5XMXDZV4TBAZAYXQA353WXDMWS7JWEWBSQ5X7LF2OPNVCQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.210108717 (60h5m35.4004073s) [noise=true , address=45.91.116.8:4045 , piece_id=5CAKK3A4QCUHXIFXHWSJBP2WRHP2GD3IDIH3GCATLUKAIXTJFXEA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.210951388 (60h5m35.40124997s) [noise=true , address=77.90.31.12:4592 , piece_id=CJGK6HYFSIJ65TZMWSTRZA6ENE2W4NQPPLJUIC564JAVQBCUFNPA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.211817916 (60h5m35.402116502s) [noise=true , address=104.36.82.5:4839 , piece_id=APYPE447JL5DG3OPPQVNFTUXWUQVBRAD72SR5EFYP5ERZW2ZCTLQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.213427607 (60h5m35.403726191s) [noise=true , address=104.36.82.13:4187 , piece_id=PXJLSCAHD6BW23Q7QI7DOZUHOSRRTYJHSMVDIYXSSXDLQGBLLMJA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.214298310 (60h5m35.4045969s) [noise=true , address=104.36.82.15:4258 , piece_id=BDSDXBWQML3AI5H5XX4GNJVIXUK57LHDPKA6W6M5UAWVC65UA5WA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.215150105 (60h5m35.405448691s) [noise=true , address=104.36.82.16:4293 , piece_id=AJOCFEAXFKAUCSDC54HEATCOJRH3VHWOWCQYS4AN4WMZESE5GGDQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.215972205 (60h5m35.406270796s) [noise=true , address=104.36.82.6:4876 , piece_id=UQHCILHS4ENQMEJDU4ZG3X2DO6GCYRYA2Y75SVOKP7YIPOJZ4OGQ, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.218143703 (60h5m35.408442287s) [noise=true , address=77.90.31.18:4805 , piece_id=H3747CGDBMFJ35PPTWGFDR6SQTW4HYAWVLWCCR3R4S4SSVE7K73Q, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.220331302 (60h5m35.410629887s) [noise=true , address=45.91.116.7:4009 , piece_id=YQ5DBOFY2CCIOV637MVJGX2BMGILHOGRVTNE52LQ22PZA2W6FLAA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.222963530 (60h5m35.413262129s) [noise=true , address=104.36.82.10:4078 , piece_id=S5VQ5IXNYD4BQBKPSZLDZTXIZKBCYMKYUHI3OYWBNLTNRH22LAYA, seg_pos={0 0}, upload=57618]: Uploading piece... Jul 26 23:41:04.818083857 (60h5m36.008382426s) [upload=57618]: Next returned nil segment Jul 26 23:41:04.818106433 (60h5m36.008405003s) [upload=57618]: Waiting for error group managing segments... Jul 26 23:41:04.959009630 (60h5m36.149308203s) [noise=true , address=45.91.116.8:4045 , piece_id=5CAKK3A4QCUHXIFXHWSJBP2WRHP2GD3IDIH3GCATLUKAIXTJFXEA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.959790625 (60h5m36.150089198s) [noise=true , address=77.90.31.12:4592 , piece_id=CJGK6HYFSIJ65TZMWSTRZA6ENE2W4NQPPLJUIC564JAVQBCUFNPA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.967045010 (60h5m36.15734359s) [noise=true , address=77.90.31.18:4807 , piece_id=WOMXZGRHNTCUIWN2GMOVLCUPK4FHMEV72RNKRZU4ACGA25ZUEF5A, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.970358136 (60h5m36.160656708s) [noise=true , address=45.91.116.13:4154 , piece_id=IHBPYX5XMXDZV4TBAZAYXQA353WXDMWS7JWEWBSQ5X7LF2OPNVCQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.973625935 (60h5m36.163924515s) [noise=true , address=77.90.31.14:4662 , piece_id=L23MCSL2BLZRHR3M3ABEJ3GBL4LOVYASILFDNKKLGSOD2WCQEXIQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.977928879 (60h5m36.168227458s) [noise=true , address=45.91.116.7:4009 , piece_id=YQ5DBOFY2CCIOV637MVJGX2BMGILHOGRVTNE52LQ22PZA2W6FLAA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:04.978183641 (60h5m36.16848221s) [noise=true , address=77.90.31.18:4805 , piece_id=H3747CGDBMFJ35PPTWGFDR6SQTW4HYAWVLWCCR3R4S4SSVE7K73Q, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:06.339880594 (60h5m37.530179165s) [noise=true , address=77.90.31.19:4843 , piece_id=33NPNN5P57MT2PNG5MPYURE6NCOZ43ZF5WF2YHBAZ3577YHSANSQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:06.365822282 (60h5m37.556120851s) [noise=true , address=77.90.31.13:4626 , piece_id=DB4H2EWGUP37SVU5EGM44O5C5AGFJ4GQJM4DMOLIE5ZPNMYNYVZQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:24.418277767 (60h5m55.608576342s) [noise=true , address=104.36.82.6:4876 , piece_id=UQHCILHS4ENQMEJDU4ZG3X2DO6GCYRYA2Y75SVOKP7YIPOJZ4OGQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: <nil> Jul 26 23:41:33.825391388 (60h6m5.015689977s) [noise=true , address=104.36.82.13:4187 , piece_id=PXJLSCAHD6BW23Q7QI7DOZUHOSRRTYJHSMVDIYXSSXDLQGBLLMJA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:54044->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:54044->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54044->104.36.82.13:4187: use of closed network connection Jul 26 23:41:33.825536938 (60h6m5.015835506s) [noise=true , address=104.36.82.5:4839 , piece_id=APYPE447JL5DG3OPPQVNFTUXWUQVBRAD72SR5EFYP5ERZW2ZCTLQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:52160->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:52160->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52160->104.36.82.5:4839: use of closed network connection Jul 26 23:41:33.825552306 (60h6m5.015850889s) [noise=true , address=104.36.82.10:4078 , piece_id=S5VQ5IXNYD4BQBKPSZLDZTXIZKBCYMKYUHI3OYWBNLTNRH22LAYA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:36622->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:36622->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36622->104.36.82.10:4078: use of closed network connection Jul 26 23:41:33.825632093 (60h6m5.015930678s) [noise=true , address=104.36.82.11:4113 , piece_id=5YLHKA2L5F75OE56XTFGDNU4BHONFFLWT2ICI7EYOOLD7LQORFYQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:38058->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:38058->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38058->104.36.82.11:4113: use of closed network connection Jul 26 23:41:33.825673432 (60h6m5.015971997s) [noise=true , address=104.36.82.15:4258 , piece_id=BDSDXBWQML3AI5H5XX4GNJVIXUK57LHDPKA6W6M5UAWVC65UA5WA, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 139.59.144.22:32910->104.36.82.15:4258: use of closed network connection; write tcp 139.59.144.22:32910->104.36.82.15:4258: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32910->104.36.82.15:4258: use of closed network connection Jul 26 23:41:33.825700719 (60h6m5.015999301s) [noise=true , address=104.36.82.16:4293 , piece_id=AJOCFEAXFKAUCSDC54HEATCOJRH3VHWOWCQYS4AN4WMZESE5GGDQ, seg_pos={0 0}, upload=57618]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:44700->104.36.82.16:4293: use of closed network connection; write tcp 139.59.144.22:44700->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44700->104.36.82.16:4293: use of closed network connection Jul 26 23:41:33.825746985 (60h6m5.016045575s) [seg_pos={0 0}, upload=57618]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:41:33.933876567 (60h6m5.124175141s) [upload=57618]: Done uploading segments... Jul 26 23:41:33.933942296 (60h6m5.12424088s) [upload=57618]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.110383986s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....X.......X........XX....X....X.X..X........XX.X....X....] duration: 36.939437098s FAIL (2025-07-26 23:41:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.939451804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........X.XX...XX..X..XXXXX....XX.XX..X.......X......X..XX] duration: 30.00105579s FAIL (2025-07-26 23:41:07 UTC) orbiter warning: context deadline exceeded Duration: 30.001074578s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [..XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X] duration: 1m0.118400544s FAIL (2025-07-26 23:40:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== pload=57600]: Got next segment. Inline: false Jul 26 23:39:59.640133629 (60h4m30.83043221s) [upload=57600]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:59.640172130 (60h4m30.830470709s) [upload=57600]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:59.836029504 (60h4m31.026328083s) [seg_pos={0 0}, upload=57600]: Begin upload segment... Jul 26 23:39:59.836088725 (60h4m31.026387294s) [seg_pos={0 0}, upload=57600]: Done begin upload segment. Jul 26 23:39:59.836122384 (60h4m31.026420962s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836155585 (60h4m31.026454174s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.836522794 (60h4m31.026821381s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.837656627 (60h4m31.027955209s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838186932 (60h4m31.028485513s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838261347 (60h4m31.02855993s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.838974935 (60h4m31.029273521s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839024475 (60h4m31.029323056s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839824134 (60h4m31.030122716s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.839893733 (60h4m31.030192318s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.840657395 (60h4m31.030955998s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841380506 (60h4m31.031679096s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.841775028 (60h4m31.03207361s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843059453 (60h4m31.033358038s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.843511089 (60h4m31.033809675s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.844586849 (60h4m31.034885436s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845046976 (60h4m31.035345569s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:39:59.845834257 (60h4m31.036132846s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Uploading piece... Jul 26 23:40:00.366525002 (60h4m31.556823573s) [noise=true , address=79.127.201.214:4321 , piece_id=ZT6SLU7P6DDWFO5I6RBB54BZZCY35MWKUG2TVEBGETGWSOJF2FJA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:00.378498951 (60h4m31.568797521s) [upload=57600]: Next returned nil segment Jul 26 23:40:00.378899577 (60h4m31.569198163s) [upload=57600]: Waiting for error group managing segments... Jul 26 23:40:00.573447807 (60h4m31.763746387s) [noise=true , address=77.90.31.15:4713 , piece_id=XZY6T2URKLGQWSLBG7TO6EU73RT5K42ORR7OGZBRVE7O6WUMPKZA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.593073616 (60h4m31.783372184s) [noise=true , address=77.90.31.12:4595 , piece_id=PNRXJQR26KWEOXJNTPKENN2ZYS4CVGG2UFEDGQDRV4ZB5JJQR7WA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.634561036 (60h4m31.824859609s) [noise=true , address=77.90.31.16:4743 , piece_id=DVFVUXCI32K3BQT5EZHYYJETAUE53CAWDAZGS45LD3HGUMZXNJ6A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:00.654282381 (60h4m31.844580956s) [noise=true , address=79.127.219.38:4013 , piece_id=6LY7YIB5Z46C3NW76A3TASYPNQUDVRPHE5BIR76ACMTO4YOAWUDQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.144531131 (60h4m32.334829714s) [noise=true , address=77.90.31.18:4821 , piece_id=TFJXOQYWF2FSAXPHJZPFTY2QCIWELFRHSYSXN2IL7D2TDNDOF7LQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.201706894 (60h4m32.39200549s) [noise=true , address=79.127.205.235:4680 , piece_id=KDNFCL5AWJ2N5BMAWAJWAP6EYXSFWQRHCZEY4BCRETMM5SV3BK5A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:01.372568543 (60h4m32.562867135s) [noise=true , address=64.49.13.138:4483 , piece_id=RESZ5QSW4EF46PMGJAGTLMT6DMEU6H5F47OEV5XGNW2NTYMXLZKA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:22.567473269 (60h4m53.757771848s) [noise=true , address=104.36.82.53:4125 , piece_id=YURC7L6ISX32YRNYPGQZEW4QHZJSYPN2XL262JPHDR2UJ2G646TA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:30.015266116 (60h5m1.205564685s) [noise=true , address=104.36.82.5:4842 , piece_id=APEUAH4ZR64NZ6SZWS2LLDHJLCAPZXW4BYPI3JDYT6LXLOWASD7A, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:32.139831734 (60h5m3.330130317s) [noise=true , address=104.36.82.17:4329 , piece_id=PV4YU7TW32MCJVD7FIGJVAY24DUUQ2MHVHYE4FE5ULSTJVE5V3DA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:41.195176201 (60h5m12.385474787s) [noise=true , address=104.36.82.54:4164 , piece_id=D3M2JEX6BWT3UQEBF22Q6PV3QFZYULV34OSX7MWIM7Z5M5DT672Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:46.287007839 (60h5m17.477306424s) [noise=true , address=104.36.82.47:4523 , piece_id=YOTYK53AQQASD3HX6LN5N7YLOSNFFSNY32YNKQYQMGOHJIEPGUUQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:48.854794351 (60h5m20.04509292s) [noise=true , address=104.36.82.11:4132 , piece_id=4RTXCYXLCJFO7ZWOU3WFDXWNCD5VEHGB2PUVA2C2343PEIDSGUYQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: <nil> Jul 26 23:40:59.366831107 (60h5m30.55712968s) [noise=true , address=104.36.82.8:4976 , piece_id=ULVGOODNQKV2ZQDKCBHUXPIDK5MJCJAD55T7HRUF2OIWJXKUBCHQ, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:129av1sWniou1aWC1EReoKyaxztX2g6bQwzSFpGqwxx3ZQ3nz5e, address:104.36.82.8:4976): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367098403 (60h5m30.557396979s) [noise=true , address=104.36.82.6:4890 , piece_id=2OOSFBKDJ4CLLYW4Y7WVF7K2JOC4EM5J26FXYUIYTS2USADO7D2Q, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:104.36.82.6:4890): protocol: expected piece hash; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; write tcp 139.59.144.22:47720->104.36.82.6:4890: use of closed network connection; EOF Jul 26 23:40:59.367280703 (60h5m30.557579272s) [noise=true , address=104.36.82.21:4501 , piece_id=M6OP7FJU5CXXR4AQGD7DXY4F4EZXL3R5CCLE4DO5B32P7SN3VSDA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38542->104.36.82.21:4501: use of closed network connection Jul 26 23:40:59.367402645 (60h5m30.557701214s) [noise=true , address=104.36.82.46:4508 , piece_id=H2VIHG23ZHRKMWEJNODCEIMOGZZ3WQ6Q4EU2BNVMZ7J6VQK3EJIA, seg_pos={0 0}, upload=57600]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:59.367432324 (60h5m30.557730904s) [seg_pos={0 0}, upload=57600]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:40:59.483361686 (60h5m30.673660258s) [upload=57600]: Done uploading segments... Jul 26 23:40:59.483390751 (60h5m30.67368933s) [upload=57600]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.118425892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [........X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXX] duration: 30.008635745s FAIL (2025-07-26 23:40:57 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.008663152s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [........X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXX] duration: 30.117386842s FAIL (2025-07-26 23:40:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Jul 26 23:40:24.122957358 (60h4m55.313255941s) [upload=57607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:40:24.122980055 (60h4m55.313278638s) [upload=57607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:40:24.305698114 (60h4m55.495996693s) [seg_pos={0 0}, upload=57607]: Begin upload segment... Jul 26 23:40:24.305766549 (60h4m55.496065141s) [seg_pos={0 0}, upload=57607]: Done begin upload segment. Jul 26 23:40:24.305802049 (60h4m55.496100632s) [noise=true , address=104.36.82.5:4837 , piece_id=B3SVZIO7JEUOEG2PUM5QYSRMVRZR2OKIITKY47LL3BGF3DZR6R5Q, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.305836971 (60h4m55.496135557s) [noise=true , address=104.36.82.8:4972 , piece_id=MVMPRVVXJX4BMEQ7PIQJ654KIEPK57HXXIBAES4MO7SXUCRPLETA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.306242474 (60h4m55.496541064s) [noise=true , address=104.36.82.2:4730 , piece_id=VGCWLTGIUGARWB2GFFENQ2QAELATNVB3GWTXQRKC6IFCONR76VCQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.306339901 (60h4m55.496638493s) [noise=true , address=45.91.116.8:4047 , piece_id=5GXCQ5GZLOTCARRQCG7WFXDQX7M7HWO463VKSDZKE757T7KVZQDA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.306642181 (60h4m55.496940754s) [noise=true , address=77.90.31.19:4841 , piece_id=6V55KIUUO4BK27RH3FE5NHSUENQUY3XNU5DJIP5M4EPNLLKECRIA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.306832953 (60h4m55.497131536s) [noise=true , address=45.91.116.5:4297 , piece_id=IVGUQGABKOWY2UNHU7YIVHQ6QE2FSV6A3SVKFBHS6NKRDY7EW2TQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.307218196 (60h4m55.497516783s) [noise=true , address=45.91.116.7:4010 , piece_id=NZ33GYR3PPPIEUHTPSCCAR42YRJNLW7HW7QPTWAAYEB6ZLXW2ZFA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.307779529 (60h4m55.498078108s) [noise=true , address=104.36.82.9:4005 , piece_id=AG5X7N662Q67USFR7JIJWQTPLWOWFBL5UBW5AEOQWXXJQUT72MWQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.308577048 (60h4m55.498875637s) [noise=true , address=104.36.82.7:4910 , piece_id=VZ4IWORKYZ5VWYFFEXUFUP3IVBBWUNUIWASM2TB3DFSZTOA5J6IA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.310140573 (60h4m55.500439162s) [noise=true , address=45.91.116.10:4082 , piece_id=HZCX3UTQLURXJZPFIFFR6IB4OF6FLKSGVWA4NYETB2VAAJTXES7A, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.310537543 (60h4m55.500836134s) [noise=true , address=104.36.82.13:4185 , piece_id=IPUYUKVJR3REPO4QKCCELJM2RKHQW2YO2HWROENWU63DMDJFM7FA, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.310865545 (60h4m55.501164135s) [noise=true , address=104.36.82.14:4223 , piece_id=5PJC2LFE5X2FBWIL7QDHRNEWJIJDRHYF7D6FOAK5WFUSODN6NOJQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.311190515 (60h4m55.501489098s) [noise=true , address=104.36.82.5:4840 , piece_id=O25PEFBTAARPNVXZMLB7TYDE4ZEGZFGXMYTPRLTDPCGBOPOK3RIQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.311507685 (60h4m55.501806352s) [noise=true , address=104.36.82.11:4115 , piece_id=3ANAZGC5YHE7WXQ7BPDFIZS6UFEU2XISJWODUGWJ2JKRR4SDXFUQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.311832380 (60h4m55.502130961s) [noise=true , address=104.36.82.16:4293 , piece_id=Z2AOPXGF2IG6JUFF47GG3AKKTVCZYHSXUANONCMJJUYO6TE7XSGQ, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.312141382 (60h4m55.50243997s) [noise=true , address=77.90.31.14:4661 , piece_id=HFASWRO2JM6LZSV7CV5HHVVRLIOWBIHYDN6V3NWCJNK4W6JYHI4Q, seg_pos={0 0}, upload=57607]: Uploading piece... Jul 26 23:40:24.916559965 (60h4m56.106858549s) [upload=57607]: Next returned nil segment Jul 26 23:40:24.916581781 (60h4m56.106880368s) [upload=57607]: Waiting for error group managing segments... Jul 26 23:40:25.058875307 (60h4m56.24917389s) [noise=true , address=45.91.116.7:4010 , piece_id=NZ33GYR3PPPIEUHTPSCCAR42YRJNLW7HW7QPTWAAYEB6ZLXW2ZFA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:25.066882708 (60h4m56.257181276s) [noise=true , address=45.91.116.10:4082 , piece_id=HZCX3UTQLURXJZPFIFFR6IB4OF6FLKSGVWA4NYETB2VAAJTXES7A, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:25.073193364 (60h4m56.263491937s) [noise=true , address=45.91.116.8:4047 , piece_id=5GXCQ5GZLOTCARRQCG7WFXDQX7M7HWO463VKSDZKE757T7KVZQDA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:25.091768365 (60h4m56.282066952s) [noise=true , address=77.90.31.14:4661 , piece_id=HFASWRO2JM6LZSV7CV5HHVVRLIOWBIHYDN6V3NWCJNK4W6JYHI4Q, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:25.171437875 (60h4m56.361736465s) [noise=true , address=45.91.116.5:4297 , piece_id=IVGUQGABKOWY2UNHU7YIVHQ6QE2FSV6A3SVKFBHS6NKRDY7EW2TQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:26.430876356 (60h4m57.621174933s) [noise=true , address=77.90.31.19:4841 , piece_id=6V55KIUUO4BK27RH3FE5NHSUENQUY3XNU5DJIP5M4EPNLLKECRIA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:34.850521057 (60h5m6.040819638s) [noise=true , address=104.36.82.7:4910 , piece_id=VZ4IWORKYZ5VWYFFEXUFUP3IVBBWUNUIWASM2TB3DFSZTOA5J6IA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:48.698197240 (60h5m19.888495819s) [noise=true , address=104.36.82.16:4293 , piece_id=Z2AOPXGF2IG6JUFF47GG3AKKTVCZYHSXUANONCMJJUYO6TE7XSGQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: <nil> Jul 26 23:40:53.852354167 (60h5m25.042652746s) [noise=true , address=104.36.82.11:4115 , piece_id=3ANAZGC5YHE7WXQ7BPDFIZS6UFEU2XISJWODUGWJ2JKRR4SDXFUQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:53.852628116 (60h5m25.042926708s) [noise=true , address=104.36.82.5:4840 , piece_id=O25PEFBTAARPNVXZMLB7TYDE4ZEGZFGXMYTPRLTDPCGBOPOK3RIQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:53.852717119 (60h5m25.043015726s) [noise=true , address=104.36.82.5:4837 , piece_id=B3SVZIO7JEUOEG2PUM5QYSRMVRZR2OKIITKY47LL3BGF3DZR6R5Q, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:53.853010209 (60h5m25.043308788s) [noise=true , address=104.36.82.2:4730 , piece_id=VGCWLTGIUGARWB2GFFENQ2QAELATNVB3GWTXQRKC6IFCONR76VCQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:40:53.853121673 (60h5m25.043420252s) [noise=true , address=104.36.82.8:4972 , piece_id=MVMPRVVXJX4BMEQ7PIQJ654KIEPK57HXXIBAES4MO7SXUCRPLETA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:37826->104.36.82.8:4972: use of closed network connection; write tcp 139.59.144.22:37826->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37826->104.36.82.8:4972: use of closed network connection Jul 26 23:40:53.853186175 (60h5m25.043484754s) [noise=true , address=104.36.82.13:4185 , piece_id=IPUYUKVJR3REPO4QKCCELJM2RKHQW2YO2HWROENWU63DMDJFM7FA, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:59178->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:59178->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59178->104.36.82.13:4185: use of closed network connection Jul 26 23:40:53.853244815 (60h5m25.043543395s) [noise=true , address=104.36.82.9:4005 , piece_id=AG5X7N662Q67USFR7JIJWQTPLWOWFBL5UBW5AEOQWXXJQUT72MWQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 139.59.144.22:37248->104.36.82.9:4005: use of closed network connection; write tcp 139.59.144.22:37248->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37248->104.36.82.9:4005: use of closed network connection Jul 26 23:40:53.853306652 (60h5m25.043605232s) [noise=true , address=104.36.82.14:4223 , piece_id=5PJC2LFE5X2FBWIL7QDHRNEWJIJDRHYF7D6FOAK5WFUSODN6NOJQ, seg_pos={0 0}, upload=57607]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:50204->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:50204->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50204->104.36.82.14:4223: use of closed network connection Jul 26 23:40:53.853323745 (60h5m25.043622324s) [seg_pos={0 0}, upload=57607]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:40:53.967655904 (60h5m25.157954476s) [upload=57607]: Done uploading segments... Jul 26 23:40:53.967688328 (60h5m25.157986907s) [upload=57607]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.117425768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......X.......X........XX....X....X.X..X........XX.X....X...] duration: 1m0.000981473s FAIL (2025-07-26 23:40:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001006998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...........X.XX...XX..X..XXXXX....XX.XX..X.......X......X..X] duration: 30.00155109s FAIL (2025-07-26 23:40:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001580174s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.........X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXX] duration: 30.107869639s FAIL (2025-07-26 23:39:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , address=45.91.116.8:4047 , piece_id=CEOFBXYN5QKVUGOZUSDHEQHFHAWFEA7N2JUS7ZCIW3CGQ52VNJVA, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.806449966 (60h3m53.996748555s) [noise=true , address=104.36.82.7:4912 , piece_id=3Z5ARX3EOF3JIYPT6AYXZBO3LJZOPZR6BVKUM6OGO75SEBR7AZIQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.806704532 (60h3m53.997003116s) [noise=true , address=104.36.82.10:4077 , piece_id=SYXVD7VTL35T4CSF6S4XYZNUKO2BDIV54XXRKHSSTPOLL3ZGASFA, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.806914690 (60h3m53.997213279s) [noise=true , address=45.91.116.5:4297 , piece_id=6RWMMTJGDXCJFPVWIBC4YILTK64YYUAHICD7UBTZHX2LLI4FPKEQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.806986895 (60h3m53.997285474s) [noise=true , address=104.36.82.9:4007 , piece_id=HEHPD5RB4H6N6CBOTBDNWCKWVJCWHULAPASKZ2RWWZ3JH2OTHI2A, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.807393383 (60h3m53.997691972s) [noise=true , address=77.90.31.14:4663 , piece_id=4HTZLBV76MW5IFSWEBRI2BJEWQTN5P772DYYAWFH3RPGG6XX644Q, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.807907788 (60h3m53.99820638s) [noise=true , address=104.36.82.13:4187 , piece_id=CWQ5TL3NOSUJA4HEFRIWHRR3ZKOR72XJYLJCOWQYINU6WSZ747AA, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.808204359 (60h3m53.998502941s) [noise=true , address=104.36.82.7:4911 , piece_id=TVWVU7WLAEQKCWWNS6T43UXPDI5WT3GWYVIYGCTZFYTYRWBX54UQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.808474666 (60h3m53.998773245s) [noise=true , address=104.36.82.6:4875 , piece_id=GPVEEXHXHAAAX2BZZ5UB2RIJFI3AWBWZGUJ4Q2IW2E2Z66KCH65Q, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.808848611 (60h3m53.999147199s) [noise=true , address=45.91.116.7:4009 , piece_id=VHZIK6TQ6BMG7OFH4CDJXTZV4DU2YNV7DRB55KPGVTX6NKVGUKFQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.809248774 (60h3m53.999547353s) [noise=true , address=45.91.116.6:4973 , piece_id=UCE6LKXJPMIBV7KCNT3WQR7Y7GM6QQTMIM6JJIXMDA6D2RUIK3VQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.809526466 (60h3m53.999825045s) [noise=true , address=45.91.116.12:4225 , piece_id=K36FPJP7HVHUOK2TPWQGSAHY4ORKCSWBQTTQOS4ANLJGY6MTNFZQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.809846782 (60h3m54.000145363s) [noise=true , address=45.91.116.4:4939 , piece_id=Q7HITJSBANSMROG4SMFRUFNCFMHL7U324AAWMJBPDKHGM47YXP2A, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:22.810334192 (60h3m54.000632781s) [noise=true , address=104.36.82.10:4078 , piece_id=NN55SJMDRNF7K56F226BLL4QPTBPTLUJAG5X6DNI2FPIULCT5DFQ, seg_pos={0 0}, upload=57591]: Uploading piece... Jul 26 23:39:23.416303426 (60h3m54.606602015s) [upload=57591]: Next returned nil segment Jul 26 23:39:23.416337766 (60h3m54.606636356s) [upload=57591]: Waiting for error group managing segments... Jul 26 23:39:23.555572418 (60h3m54.745870998s) [noise=true , address=45.91.116.4:4939 , piece_id=Q7HITJSBANSMROG4SMFRUFNCFMHL7U324AAWMJBPDKHGM47YXP2A, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.558184535 (60h3m54.748483115s) [noise=true , address=77.90.31.14:4663 , piece_id=4HTZLBV76MW5IFSWEBRI2BJEWQTN5P772DYYAWFH3RPGG6XX644Q, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.563428269 (60h3m54.753726849s) [noise=true , address=45.91.116.6:4973 , piece_id=UCE6LKXJPMIBV7KCNT3WQR7Y7GM6QQTMIM6JJIXMDA6D2RUIK3VQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.566241819 (60h3m54.756540399s) [noise=true , address=45.91.116.8:4047 , piece_id=CEOFBXYN5QKVUGOZUSDHEQHFHAWFEA7N2JUS7ZCIW3CGQ52VNJVA, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.571921800 (60h3m54.762220381s) [noise=true , address=45.91.116.12:4225 , piece_id=K36FPJP7HVHUOK2TPWQGSAHY4ORKCSWBQTTQOS4ANLJGY6MTNFZQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.591938601 (60h3m54.782237183s) [noise=true , address=45.91.116.7:4009 , piece_id=VHZIK6TQ6BMG7OFH4CDJXTZV4DU2YNV7DRB55KPGVTX6NKVGUKFQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:23.711751682 (60h3m54.902050248s) [noise=true , address=45.91.116.5:4297 , piece_id=6RWMMTJGDXCJFPVWIBC4YILTK64YYUAHICD7UBTZHX2LLI4FPKEQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: <nil> Jul 26 23:39:52.428010654 (60h4m23.61830924s) [noise=true , address=104.36.82.13:4187 , piece_id=CWQ5TL3NOSUJA4HEFRIWHRR3ZKOR72XJYLJCOWQYINU6WSZ747AA, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:39:52.428211663 (60h4m23.618510242s) [noise=true , address=104.36.82.10:4078 , piece_id=NN55SJMDRNF7K56F226BLL4QPTBPTLUJAG5X6DNI2FPIULCT5DFQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:52188->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:52188->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52188->104.36.82.10:4078: use of closed network connection Jul 26 23:39:52.428230270 (60h4m23.618528849s) [noise=true , address=104.36.82.6:4875 , piece_id=GPVEEXHXHAAAX2BZZ5UB2RIJFI3AWBWZGUJ4Q2IW2E2Z66KCH65Q, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 139.59.144.22:33064->104.36.82.6:4875: use of closed network connection; write tcp 139.59.144.22:33064->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33064->104.36.82.6:4875: use of closed network connection Jul 26 23:39:52.428310689 (60h4m23.618609269s) [noise=true , address=104.36.82.7:4912 , piece_id=3Z5ARX3EOF3JIYPT6AYXZBO3LJZOPZR6BVKUM6OGO75SEBR7AZIQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 139.59.144.22:56240->104.36.82.7:4912: use of closed network connection; write tcp 139.59.144.22:56240->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56240->104.36.82.7:4912: use of closed network connection Jul 26 23:39:52.428342232 (60h4m23.618640801s) [noise=true , address=104.36.82.16:4293 , piece_id=O2HAUTCTW4FCM3CGP4Z7I7PF4WF4TVZTKO2XXO3XFMNRKZ4ZKK4A, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:51200->104.36.82.16:4293: use of closed network connection; write tcp 139.59.144.22:51200->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51200->104.36.82.16:4293: use of closed network connection Jul 26 23:39:52.428411355 (60h4m23.618709924s) [noise=true , address=104.36.82.7:4911 , piece_id=TVWVU7WLAEQKCWWNS6T43UXPDI5WT3GWYVIYGCTZFYTYRWBX54UQ, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 139.59.144.22:37608->104.36.82.7:4911: use of closed network connection; write tcp 139.59.144.22:37608->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37608->104.36.82.7:4911: use of closed network connection Jul 26 23:39:52.428453744 (60h4m23.618752323s) [noise=true , address=104.36.82.4:4803 , piece_id=CQYOP2B5ZATAJEZZVWSTICLVUAG3XEGJK3FKMFRAJTGEVZDLSI4A, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58476->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58476->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58476->104.36.82.4:4803: use of closed network connection Jul 26 23:39:52.428481419 (60h4m23.618779998s) [noise=true , address=104.36.82.9:4007 , piece_id=HEHPD5RB4H6N6CBOTBDNWCKWVJCWHULAPASKZ2RWWZ3JH2OTHI2A, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:51570->104.36.82.9:4007: use of closed network connection; context deadline exceeded Jul 26 23:39:52.428519854 (60h4m23.618818422s) [noise=true , address=104.36.82.10:4077 , piece_id=SYXVD7VTL35T4CSF6S4XYZNUKO2BDIV54XXRKHSSTPOLL3ZGASFA, seg_pos={0 0}, upload=57591]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 139.59.144.22:46204->104.36.82.10:4077: use of closed network connection; write tcp 139.59.144.22:46204->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46204->104.36.82.10:4077: use of closed network connection Jul 26 23:39:52.428568101 (60h4m23.618866684s) [seg_pos={0 0}, upload=57591]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:39:52.534549780 (60h4m23.724848348s) [upload=57591]: Done uploading segments... Jul 26 23:39:52.534576412 (60h4m23.724874991s) [upload=57591]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.107896803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [.........X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XX] duration: 30.143869649s FAIL (2025-07-26 23:39:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.143896496s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......X.......X........XX....X....X.X..X........XX.X....X..] duration: 44.45758584s FAIL (2025-07-26 23:39:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.457656937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [...XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......] duration: 50.215647401s FAIL (2025-07-26 23:39:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Inline: false Jul 26 23:33:01.730895780 (59h57m32.921194363s) [upload=57493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:33:01.730902773 (59h57m32.921201361s) [upload=57493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:33:01.938820211 (59h57m33.129118793s) [seg_pos={0 0}, upload=57493]: Begin upload segment... Jul 26 23:33:01.938930034 (59h57m33.129228617s) [seg_pos={0 0}, upload=57493]: Done begin upload segment. Jul 26 23:33:01.939000134 (59h57m33.129298707s) [noise=true , address=104.36.82.35:4597 , piece_id=B55GVM5M7O7YAZWHKO5X4FB5BIIH5L2IWOOVAS4WVWXSLOL7GOIQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.940029976 (59h57m33.130328557s) [noise=true , address=77.90.31.19:4860 , piece_id=HVSVVJIZQL5EXGSFR6R36E3RVY4MAW2EIFZSBCUSBCPAVPNXCY2A, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.940985291 (59h57m33.131283873s) [noise=true , address=77.90.31.11:4581 , piece_id=C7SD5R2X3SRJ7LK7HF2DPT6GNCXFC4K4DPDLHCYPSSDTYSVMJ7WQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.941809183 (59h57m33.132107758s) [noise=true , address=77.90.31.13:4651 , piece_id=FTZJRCIFIWGFRO6OCOBXIWHA7VBLSS27YVKJHBZWEZ3LN53RSWXQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.942536770 (59h57m33.132835357s) [noise=true , address=79.127.201.214:4321 , piece_id=HE77CVCTM6KBQPE6WZRSTZQXGVG4HLUYFNIKJONEYAHZ2AHY7OPQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.943208143 (59h57m33.133506724s) [noise=true , address=79.127.201.215:4351 , piece_id=ZTEAICY2G3LTAZFF7P6LMUIXXI7FDCH3ZMFJWDXNMPEDRYEFHYHQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.943760642 (59h57m33.134059234s) [noise=true , address=77.90.31.10:4540 , piece_id=YV5BRFWAMNAWZ2UMR37RLKYQZJKVZRAESET43AQZXSLO55LVF3LQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.944330692 (59h57m33.134629284s) [noise=true , address=77.90.31.16:4766 , piece_id=2V2HOCPOQ7LBY3Z4JZGK3OKRZ4NV7D73L3H6PYEJW65CAORBSIDQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.945063463 (59h57m33.135362055s) [noise=true , address=77.90.31.18:4821 , piece_id=AUSIVCYCNYRFNO5TZZ6N5735BH7XZUJRZGPZLO24KG6O6OLSS7TQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.946139284 (59h57m33.136437874s) [noise=true , address=104.36.82.55:4198 , piece_id=F7OWPBEQG5XVMFXWVW5MXFA3WS22X6MAXUO7LRGUAPO5KDAE2KVA, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.946635141 (59h57m33.136933731s) [noise=true , address=104.36.82.52:4095 , piece_id=UKL56JS27RB7J733BJ3JFVI5ETBW44MT4MMERVYYNVOWC7QE627Q, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.947427537 (59h57m33.137726117s) [noise=true , address=104.36.82.46:4489 , piece_id=NFGB4WCTZGV4MLWJB76QEV6D3FMBSRPBNBKIRNFG2PMHQNNFPXWA, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.947978131 (59h57m33.13827672s) [noise=true , address=79.127.220.101:4048 , piece_id=DIIXEGOODKJKMT2SSJF7FSAYQJLL62Y4PXS6QP2LLTEAKB2ODJUA, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.948624930 (59h57m33.138923521s) [noise=true , address=104.36.82.47:4538 , piece_id=3A52OIZRDERE5GJA6C2HJRPOTJIYSKELA4XCV3ZN54SMWIJHQPTA, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.949153028 (59h57m33.139451619s) [noise=true , address=45.91.116.14:4194 , piece_id=OUQPBAH4WWDBPYXRA6VGNH2T5DP3GDN42ECMR6QSZGYD37UM5YAA, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.949830892 (59h57m33.14012948s) [noise=true , address=104.36.82.53:4135 , piece_id=S2NIIP5CZHSIEMCH4NM55GJAKO2ZV2RVWNV3632MCLOTL6VZCSVQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.950455825 (59h57m33.140754411s) [noise=true , address=79.127.205.235:4687 , piece_id=SHQVVXQVYLIDSUIA6NUSNRNXKZWMPIFXT42GSDJ7MTRHHPDD5UZQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:01.951005675 (59h57m33.141304258s) [noise=true , address=79.127.219.38:4001 , piece_id=2DOEQYQI2DH6A2RHMWZO5OZRL2RXR5PSAZM5KLM5UN7ONYUQDDYQ, seg_pos={0 0}, upload=57493]: Uploading piece... Jul 26 23:33:02.455030529 (59h57m33.645329115s) [noise=true , address=79.127.220.101:4048 , piece_id=DIIXEGOODKJKMT2SSJF7FSAYQJLL62Y4PXS6QP2LLTEAKB2ODJUA, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:02.533622581 (59h57m33.723921147s) [upload=57493]: Next returned nil segment Jul 26 23:33:02.533688316 (59h57m33.723986906s) [upload=57493]: Waiting for error group managing segments... Jul 26 23:33:02.698446976 (59h57m33.888745562s) [noise=true , address=79.127.219.38:4001 , piece_id=2DOEQYQI2DH6A2RHMWZO5OZRL2RXR5PSAZM5KLM5UN7ONYUQDDYQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:02.705182048 (59h57m33.895480635s) [noise=true , address=77.90.31.16:4766 , piece_id=2V2HOCPOQ7LBY3Z4JZGK3OKRZ4NV7D73L3H6PYEJW65CAORBSIDQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:02.730528547 (59h57m33.920827132s) [noise=true , address=77.90.31.10:4540 , piece_id=YV5BRFWAMNAWZ2UMR37RLKYQZJKVZRAESET43AQZXSLO55LVF3LQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:03.249516153 (59h57m34.439814735s) [noise=true , address=79.127.205.235:4687 , piece_id=SHQVVXQVYLIDSUIA6NUSNRNXKZWMPIFXT42GSDJ7MTRHHPDD5UZQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:04.540260194 (59h57m35.730558774s) [noise=true , address=77.90.31.13:4651 , piece_id=FTZJRCIFIWGFRO6OCOBXIWHA7VBLSS27YVKJHBZWEZ3LN53RSWXQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:06.674592944 (59h57m37.864891533s) [noise=true , address=79.127.201.214:4321 , piece_id=HE77CVCTM6KBQPE6WZRSTZQXGVG4HLUYFNIKJONEYAHZ2AHY7OPQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:15.684426245 (59h57m46.874724814s) [noise=true , address=45.91.116.14:4194 , piece_id=OUQPBAH4WWDBPYXRA6VGNH2T5DP3GDN42ECMR6QSZGYD37UM5YAA, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:18.885387861 (59h57m50.075686433s) [noise=true , address=104.36.82.52:4095 , piece_id=UKL56JS27RB7J733BJ3JFVI5ETBW44MT4MMERVYYNVOWC7QE627Q, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:20.139183001 (59h57m51.329481585s) [noise=true , address=77.90.31.19:4860 , piece_id=HVSVVJIZQL5EXGSFR6R36E3RVY4MAW2EIFZSBCUSBCPAVPNXCY2A, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:22.562777558 (59h57m53.753076141s) [noise=true , address=104.36.82.35:4597 , piece_id=B55GVM5M7O7YAZWHKO5X4FB5BIIH5L2IWOOVAS4WVWXSLOL7GOIQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:24.524016773 (59h57m55.714315341s) [noise=true , address=104.36.82.55:4198 , piece_id=F7OWPBEQG5XVMFXWVW5MXFA3WS22X6MAXUO7LRGUAPO5KDAE2KVA, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:33:37.120042335 (59h58m8.310340905s) [noise=true , address=79.127.201.215:4351 , piece_id=ZTEAICY2G3LTAZFF7P6LMUIXXI7FDCH3ZMFJWDXNMPEDRYEFHYHQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: <nil> Jul 26 23:34:01.453134007 (59h58m32.643432586s) [noise=true , address=104.36.82.47:4538 , piece_id=3A52OIZRDERE5GJA6C2HJRPOTJIYSKELA4XCV3ZN54SMWIJHQPTA, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:34:01.453569323 (59h58m32.643867903s) [noise=true , address=77.90.31.18:4821 , piece_id=AUSIVCYCNYRFNO5TZZ6N5735BH7XZUJRZGPZLO24KG6O6OLSS7TQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:01.453682272 (59h58m32.643980838s) [noise=true , address=104.36.82.46:4489 , piece_id=NFGB4WCTZGV4MLWJB76QEV6D3FMBSRPBNBKIRNFG2PMHQNNFPXWA, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:34:01.453765602 (59h58m32.644064185s) [noise=true , address=77.90.31.11:4581 , piece_id=C7SD5R2X3SRJ7LK7HF2DPT6GNCXFC4K4DPDLHCYPSSDTYSVMJ7WQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: rpc: dial tcp 77.90.31.11:4581: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:01.453785641 (59h58m32.644084215s) [noise=true , address=104.36.82.53:4135 , piece_id=S2NIIP5CZHSIEMCH4NM55GJAKO2ZV2RVWNV3632MCLOTL6VZCSVQ, seg_pos={0 0}, upload=57493]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:50636->104.36.82.53:4135: use of closed network connection; write tcp 139.59.144.22:50636->104.36.82.53:4135: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50636->104.36.82.53:4135: use of closed network connection Jul 26 23:34:01.453819535 (59h58m32.644118119s) [seg_pos={0 0}, upload=57493]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:34:01.561814741 (59h58m32.752113318s) [upload=57493]: Done uploading segments... Jul 26 23:34:01.561848414 (59h58m32.752146995s) [upload=57493]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 50.215669919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [..........X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXX] duration: 30.104800006s FAIL (2025-07-26 23:39:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 23:38:49.648101895 (60h3m20.838400461s) [upload=57583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:49.844670994 (60h3m21.034969608s) [seg_pos={0 0}, upload=57583]: Begin upload segment... Jul 26 23:38:49.844843247 (60h3m21.035141815s) [seg_pos={0 0}, upload=57583]: Done begin upload segment. Jul 26 23:38:49.844887672 (60h3m21.035186261s) [noise=true , address=77.90.31.18:4805 , piece_id=Y3JMFJRQWWH2PDLDASNRU3VLCZBX5AZ2NODQB46QQRKYGE6UG22A, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.844939908 (60h3m21.03523849s) [noise=true , address=104.36.82.6:4873 , piece_id=5FJ4VKA6CU3Y7GABA24OW3OZRIU6IDD2JZDOS54TRLFQRPJLWWHQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.845355689 (60h3m21.035654269s) [noise=true , address=104.36.82.6:4875 , piece_id=ULEPAOHOCUV4B3P6OQYS2H3YPF3YAM4T3MLKWCHOUD3BT2TK3ACA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.845588466 (60h3m21.035887056s) [noise=true , address=104.36.82.5:4837 , piece_id=JFX42MDSOXCUGJAUS5ZURHPFVOYZSWUFZQ7AQJWOVRV2EV7HZW2Q, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.845679016 (60h3m21.035977601s) [noise=true , address=104.36.82.3:4767 , piece_id=6SYLIZMI7HRU6SX2KLFW2ACASWWC2UAWZZUVVUV4NQRHPM2TMOWQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.846109016 (60h3m21.036407597s) [noise=true , address=45.91.116.4:4939 , piece_id=4CIZQLDQMFXVILO762FYWMYRZYBIKS6VHJS5VNXPOALBINER5FFQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.846181039 (60h3m21.03647963s) [noise=true , address=45.91.116.10:4083 , piece_id=CGHXLRBFTNJGELVBSJV4Y467V6VWSUL2QXQGYZLYZ6UM5IGXJVVQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.847742058 (60h3m21.038040642s) [noise=true , address=77.90.31.19:4843 , piece_id=OSZLNARGJYK6BWBNIYCYA72ZJ3NPKJYMOIMRW2CDX5XXDGLLPMLA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.848126428 (60h3m21.038425008s) [noise=true , address=45.91.116.7:4010 , piece_id=VMNIRRKESTFM4AD2L52NTA446EAJOGUYLZS22EV3BUHVBJSR55ZQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.849224443 (60h3m21.039523028s) [noise=true , address=104.36.82.11:4113 , piece_id=KFSXGQBJ6VB2BYBGCIR7L44J7AJAHKIF2UUCPGMKZPPCFLV7OQXA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.850051515 (60h3m21.040350087s) [noise=true , address=45.91.116.5:4299 , piece_id=RADDSOYJ2XZXLBTRTUJ4YBDHKUCGC3DZNTGTPH56QT4J3IZCDTLA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.853338799 (60h3m21.043637389s) [noise=true , address=77.90.31.19:4841 , piece_id=U4YZ36TFHKPYM66OY4AWSI7A7KNYUVD2GN5TQSSVIHBDURBV6EZA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.856922199 (60h3m21.047220772s) [noise=true , address=104.36.82.5:4839 , piece_id=PATGZH6HP4X3OJ5KR3YN5LNHNVGO5DTTELZ4PG2XC5HFZ4UBUEAA, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.857811204 (60h3m21.048109776s) [noise=true , address=104.36.82.13:4185 , piece_id=ELERDHSI7JOMRT5VF7YZNTX6KXFSJDKHBYFEVULB7RN65BAYFRFQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.858392058 (60h3m21.048690628s) [noise=true , address=45.91.116.8:4048 , piece_id=4OEZIZXSPAL5TAXWHRHPRF6BK2HT46KPMMR2RJ2UAYBRVICWHVYQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:49.859044050 (60h3m21.049342623s) [noise=true , address=104.36.82.10:4078 , piece_id=WCKLG6GDYK5AGMSLNFYPTKX7EHO22Q5F7K7BOHJFXCTZAIAEUMUQ, seg_pos={0 0}, upload=57583]: Uploading piece... Jul 26 23:38:50.453135664 (60h3m21.643434234s) [upload=57583]: Next returned nil segment Jul 26 23:38:50.453170825 (60h3m21.643469404s) [upload=57583]: Waiting for error group managing segments... Jul 26 23:38:50.587324818 (60h3m21.777623399s) [noise=true , address=45.91.116.4:4939 , piece_id=4CIZQLDQMFXVILO762FYWMYRZYBIKS6VHJS5VNXPOALBINER5FFQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:50.591861019 (60h3m21.782159603s) [noise=true , address=45.91.116.7:4010 , piece_id=VMNIRRKESTFM4AD2L52NTA446EAJOGUYLZS22EV3BUHVBJSR55ZQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:50.597763317 (60h3m21.788061903s) [noise=true , address=45.91.116.8:4048 , piece_id=4OEZIZXSPAL5TAXWHRHPRF6BK2HT46KPMMR2RJ2UAYBRVICWHVYQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:50.602831707 (60h3m21.793130296s) [noise=true , address=77.90.31.18:4805 , piece_id=Y3JMFJRQWWH2PDLDASNRU3VLCZBX5AZ2NODQB46QQRKYGE6UG22A, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:50.608293164 (60h3m21.798591738s) [noise=true , address=45.91.116.10:4083 , piece_id=CGHXLRBFTNJGELVBSJV4Y467V6VWSUL2QXQGYZLYZ6UM5IGXJVVQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:50.749155329 (60h3m21.939453909s) [noise=true , address=45.91.116.5:4299 , piece_id=RADDSOYJ2XZXLBTRTUJ4YBDHKUCGC3DZNTGTPH56QT4J3IZCDTLA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:52.167961104 (60h3m23.358259671s) [noise=true , address=77.90.31.19:4843 , piece_id=OSZLNARGJYK6BWBNIYCYA72ZJ3NPKJYMOIMRW2CDX5XXDGLLPMLA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:52.190152324 (60h3m23.380450904s) [noise=true , address=77.90.31.19:4841 , piece_id=U4YZ36TFHKPYM66OY4AWSI7A7KNYUVD2GN5TQSSVIHBDURBV6EZA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:59.525452370 (60h3m30.715750955s) [noise=true , address=104.36.82.6:4875 , piece_id=ULEPAOHOCUV4B3P6OQYS2H3YPF3YAM4T3MLKWCHOUD3BT2TK3ACA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:38:59.527412846 (60h3m30.717711419s) [noise=true , address=104.36.82.6:4873 , piece_id=5FJ4VKA6CU3Y7GABA24OW3OZRIU6IDD2JZDOS54TRLFQRPJLWWHQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: <nil> Jul 26 23:39:19.379955978 (60h3m50.570254548s) [noise=true , address=104.36.82.11:4113 , piece_id=KFSXGQBJ6VB2BYBGCIR7L44J7AJAHKIF2UUCPGMKZPPCFLV7OQXA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:38252->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:38252->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38252->104.36.82.11:4113: use of closed network connection Jul 26 23:39:19.380050757 (60h3m50.570349337s) [noise=true , address=104.36.82.5:4837 , piece_id=JFX42MDSOXCUGJAUS5ZURHPFVOYZSWUFZQ7AQJWOVRV2EV7HZW2Q, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:35202->104.36.82.5:4837: use of closed network connection; write tcp 139.59.144.22:35202->104.36.82.5:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35202->104.36.82.5:4837: use of closed network connection Jul 26 23:39:19.380078460 (60h3m50.57037703s) [noise=true , address=104.36.82.10:4078 , piece_id=WCKLG6GDYK5AGMSLNFYPTKX7EHO22Q5F7K7BOHJFXCTZAIAEUMUQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:49920->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:49920->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49920->104.36.82.10:4078: use of closed network connection Jul 26 23:39:19.380132722 (60h3m50.57043129s) [noise=true , address=104.36.82.5:4839 , piece_id=PATGZH6HP4X3OJ5KR3YN5LNHNVGO5DTTELZ4PG2XC5HFZ4UBUEAA, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:33300->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:33300->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33300->104.36.82.5:4839: use of closed network connection Jul 26 23:39:19.380140205 (60h3m50.570438784s) [noise=true , address=104.36.82.13:4185 , piece_id=ELERDHSI7JOMRT5VF7YZNTX6KXFSJDKHBYFEVULB7RN65BAYFRFQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:47766->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:47766->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47766->104.36.82.13:4185: use of closed network connection Jul 26 23:39:19.380177381 (60h3m50.570475962s) [noise=true , address=104.36.82.3:4767 , piece_id=6SYLIZMI7HRU6SX2KLFW2ACASWWC2UAWZZUVVUV4NQRHPM2TMOWQ, seg_pos={0 0}, upload=57583]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:33352->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:33352->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33352->104.36.82.3:4767: use of closed network connection Jul 26 23:39:19.380207833 (60h3m50.570506412s) [seg_pos={0 0}, upload=57583]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:39:19.482434121 (60h3m50.672732764s) [upload=57583]: Done uploading segments... Jul 26 23:39:19.482497635 (60h3m50.672796201s) [upload=57583]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.104821234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [..........X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.X] duration: 30.107325381s FAIL (2025-07-26 23:39:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.10743667s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [............X.XX...XX..X..XXXXX....XX.XX..X.......X......X..] duration: 4.713285956s FAIL (2025-07-26 23:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.713334902s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)