Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................X...........................X.X.......X...] duration: 49.484349445s FAIL (2025-07-09 20:13:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.484370315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X............X...................................] duration: 2.632199495s FAIL (2025-07-09 20:13:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.632222039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 13.832081994s FAIL (2025-07-09 20:13:22 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.83211045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X............X..................................] duration: 3.091972933s FAIL (2025-07-09 20:13:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.092085739s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................X...........................X.X.......X..] duration: 36.695066678s FAIL (2025-07-09 20:12:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.695090298s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X............X.................................] duration: 9.954256282s FAIL (2025-07-09 20:12:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.954281361s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............X................................] duration: 2.604543457s FAIL (2025-07-09 20:12:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.604561094s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.868532743s FAIL (2025-07-09 20:12:19 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 2.868564434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X............X...............................] duration: 2.775885194s FAIL (2025-07-09 20:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.776017411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 33.365961268s FAIL (2025-07-09 20:11:48 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 33.366053272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................X...........................X.X.......X.] duration: 24.080521298s FAIL (2025-07-09 20:11:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.080537171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...........................X.X.......X] duration: 1m0.000847621s FAIL (2025-07-09 20:11:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000860437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 13.428785498s FAIL (2025-07-09 20:11:02 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.428821985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X............X..............................] duration: 2.8087018s FAIL (2025-07-09 20:10:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.808729799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.164863583s FAIL (2025-07-09 20:10:14 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.164898969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X...........................X.X.......] duration: 53.850078145s FAIL (2025-07-09 20:10:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.850106059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 31.347304597s FAIL (2025-07-09 20:09:56 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 31.347333565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X............X.............................] duration: 2.971948876s FAIL (2025-07-09 20:09:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.97197881s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 47.718611468s FAIL (2025-07-09 20:09:25 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 47.718652463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...........................X.X......] duration: 30.508963219s FAIL (2025-07-09 20:08:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.508986812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 34.46422389s FAIL (2025-07-09 20:08:37 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 34.464273834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X............X............................] duration: 2.682282124s FAIL (2025-07-09 20:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.682325422s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X...........................X.X.....] duration: 33.44836098s FAIL (2025-07-09 20:07:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.448391594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X............X...........................] duration: 2.544238541s FAIL (2025-07-09 20:07:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.544273425s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X............X..........................] duration: 2.864742412s FAIL (2025-07-09 20:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.864768385s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 13.312951026s FAIL (2025-07-09 20:07:06 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.312969982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...........................X.X....] duration: 59.979933204s FAIL (2025-07-09 20:06:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.979953074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 36.188024734s FAIL (2025-07-09 20:06:53 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 36.188045641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X............X.........................] duration: 2.71563661s FAIL (2025-07-09 20:06:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.715658216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 25.05664832s FAIL (2025-07-09 20:06:16 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 25.056679511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X............X........................] duration: 2.580837028s FAIL (2025-07-09 20:06:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.580867056s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...........................X.X...] duration: 38.758068346s FAIL (2025-07-09 20:05:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.758099426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X............X.......................] duration: 2.63767867s FAIL (2025-07-09 20:05:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.637708585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 14.135541096s FAIL (2025-07-09 20:05:12 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 14.135605939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X...........................X.X..] duration: 27.346505727s FAIL (2025-07-09 20:04:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.346526394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X............X......................] duration: 2.614532828s FAIL (2025-07-09 20:04:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.614561952s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.114888751s FAIL (2025-07-09 20:04:26 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 15.114917817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X...........................X.X.] duration: 41.530158394s FAIL (2025-07-09 20:04:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.530176526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X............X.....................] duration: 3.210441061s FAIL (2025-07-09 20:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.210472377s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.669393312s FAIL (2025-07-09 20:03:47 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 11.669418532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................X...........................X.X] duration: 1m0.000885655s FAIL (2025-07-09 20:03:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000898959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............X....................] duration: 3.648397025s FAIL (2025-07-09 20:03:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.64845977s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.628671884s FAIL (2025-07-09 20:02:49 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 5.628696462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............X...................] duration: 2.781265899s FAIL (2025-07-09 20:02:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781308492s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........................X.] duration: 21.941415682s FAIL (2025-07-09 20:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.941432609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 34.877496363s FAIL (2025-07-09 20:02:26 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 34.877523777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........................X] duration: 1m0.000752754s FAIL (2025-07-09 20:02:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000775746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X............X..................] duration: 3.066703772s FAIL (2025-07-09 20:02:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.066736628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 19.481057441s FAIL (2025-07-09 20:01:26 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 19.481090835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............X.................] duration: 21.413615933s FAIL (2025-07-09 20:01:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.413687558s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)