Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 32.383145239s FAIL (2025-07-09 20:26:13 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 32.383173435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X...........................X.X.......X..................] duration: 21.782466551s FAIL (2025-07-09 20:25:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.782499216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................................X.X..............] duration: 2.90267964s FAIL (2025-07-09 20:25:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.902702757s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 29.830737262s FAIL (2025-07-09 20:25:36 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 29.830801035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...................................X.X.............] duration: 2.67513074s FAIL (2025-07-09 20:25:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.675199363s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 26.114710185s FAIL (2025-07-09 20:25:06 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 26.114748056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...........................X.X.......X.................] duration: 31.121688173s FAIL (2025-07-09 20:25:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.12170506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................................X.X............] duration: 4.558864853s FAIL (2025-07-09 20:24:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.558890428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 31.186760139s FAIL (2025-07-09 20:24:40 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 31.186799949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........................X.X.......X................] duration: 28.00011472s FAIL (2025-07-09 20:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.000144002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...................................X.X...........] duration: 11.560711368s FAIL (2025-07-09 20:24:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.560736438s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 42.652352369s FAIL (2025-07-09 20:24:08 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 42.652386804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................................X.X..........] duration: 25.527166711s FAIL (2025-07-09 20:23:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.527184697s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...........................X.X.......X...............] duration: 25.30079828s FAIL (2025-07-09 20:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.300820767s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........................X.X.......X..............] duration: 25.806920426s FAIL (2025-07-09 20:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.806968031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 22.791489985s FAIL (2025-07-09 20:23:11 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 22.791534195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 32.174221645s FAIL (2025-07-09 20:22:43 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 32.174315182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...................................X.X.........] duration: 2.78691056s FAIL (2025-07-09 20:22:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786975189s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...........................X.X.......X.............] duration: 7.237090012s FAIL (2025-07-09 20:22:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.237108545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 13.527282108s FAIL (2025-07-09 20:21:58 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.527316083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........................X.X.......X............] duration: 58.864907143s FAIL (2025-07-09 20:21:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.864944282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X............X...................................X.X........] duration: 2.897799743s FAIL (2025-07-09 20:21:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.897826195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 13.020362187s FAIL (2025-07-09 20:20:49 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 13.020395136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X............X...................................X.X.......] duration: 2.738008335s FAIL (2025-07-09 20:20:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738029817s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...........................X.X.......X...........] duration: 29.791804746s FAIL (2025-07-09 20:20:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.791828162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 14.774447146s FAIL (2025-07-09 20:19:59 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 14.774474101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X............X...................................X.X......] duration: 3.122202831s FAIL (2025-07-09 20:19:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.122230737s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X............X...................................X.X.....] duration: 14.645223622s FAIL (2025-07-09 20:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.645248067s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X...........................X.X.......X..........] duration: 15.866234748s FAIL (2025-07-09 20:19:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.866257173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 17.832386655s FAIL (2025-07-09 20:19:20 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 17.832418733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...........................X.X.......X.........] duration: 29.717375436s FAIL (2025-07-09 20:19:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.717400102s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X............X...................................X.X....] duration: 2.826037964s FAIL (2025-07-09 20:18:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.826068554s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...........................X.X.......X........] duration: 44.748212686s FAIL (2025-07-09 20:18:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.748227454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 28.479228028s FAIL (2025-07-09 20:18:23 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 28.47925839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X............X...................................X.X...] duration: 2.798886897s FAIL (2025-07-09 20:17:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.79898188s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...........................X.X.......X.......] duration: 51.75532857s FAIL (2025-07-09 20:17:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.755354899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 25.555931451s FAIL (2025-07-09 20:17:17 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 25.555962032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X............X...................................X.X..] duration: 2.881892192s FAIL (2025-07-09 20:17:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.882031266s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 26.341553197s FAIL (2025-07-09 20:16:51 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 26.34158046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...........................X.X.......X......] duration: 40.703948628s FAIL (2025-07-09 20:16:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.703978992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X............X...................................X.X.] duration: 2.892777873s FAIL (2025-07-09 20:16:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.892799682s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 31.617507272s FAIL (2025-07-09 20:16:25 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 31.617535033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X............X...................................X.X] duration: 30.000428139s FAIL (2025-07-09 20:16:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000503938s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 19.673320548s FAIL (2025-07-09 20:15:53 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 19.673349815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X............X...................................X.] duration: 2.701097163s FAIL (2025-07-09 20:15:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701124799s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...........................X.X.......X.....] duration: 19.164504568s FAIL (2025-07-09 20:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.164528432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 27.092931318s FAIL (2025-07-09 20:14:56 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 27.092963762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X...........................X.X.......X....] duration: 52.137405722s FAIL (2025-07-09 20:14:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.137427721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X............X...................................X] duration: 30.001095729s FAIL (2025-07-09 20:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0011195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 31.458029077s FAIL (2025-07-09 20:14:29 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07.775154767 (394h29m7.658568397s) [upload=315464]: Got next segment. Inline: false Jul 9 19:40:07.775222675 (394h29m7.658636302s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:07.775228517 (394h29m7.658642145s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:08.077785366 (394h29m7.961199002s) [seg_pos={0 0}, upload=315464]: Begin upload segment... Jul 9 19:40:08.077860847 (394h29m7.961274483s) [seg_pos={0 0}, upload=315464]: Done begin upload segment. Jul 9 19:40:08.077958256 (394h29m7.961371896s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.077986061 (394h29m7.961399693s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.078965751 (394h29m7.962379387s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079037038 (394h29m7.962450674s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.079207315 (394h29m7.962620951s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080140241 (394h29m7.963553877s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.080378066 (394h29m7.963791701s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081072970 (394h29m7.964486607s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081177519 (394h29m7.964591157s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.081889239 (394h29m7.965302875s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082087416 (394h29m7.965501052s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.082453979 (394h29m7.965867615s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083004219 (394h29m7.966417856s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.083371902 (394h29m7.966785539s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084191128 (394h29m7.967604765s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084571607 (394h29m7.967985244s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.084912151 (394h29m7.968325788s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.085249687 (394h29m7.968663324s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Uploading piece... Jul 9 19:40:08.616577029 (394h29m8.499990646s) [noise=true , address=79.127.201.214:4306 , piece_id=U5DCBRADG2OHKAOKUL6FOUHFVYI2XRXUWXDFNBET4URYOLC5SHNQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: ecclient: upload failed (node:122DJmb292J722vUUNey4itWFfZNT35ruZpjz4awcfcU3QEmQLc, address:79.127.201.214:4306): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:08.660838400 (394h29m8.544252021s) [upload=315464]: Next returned nil segment Jul 9 19:40:08.660853085 (394h29m8.544266704s) [upload=315464]: Waiting for error group managing segments... Jul 9 19:40:08.800543989 (394h29m8.68395762s) [noise=true , address=77.90.31.15:4718 , piece_id=Y3FJTNURRRK4AUVWGEXIAVYPBLCMT5WS6DN3KNJONBXAYVRU2WKQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.802597777 (394h29m8.686011397s) [noise=true , address=77.90.31.10:4531 , piece_id=KOVJIOT5GELWIFQWAZMX6HMFIADYSL7TCHHBCDQZHDJCP6HCIZPA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.805684069 (394h29m8.689097688s) [noise=true , address=77.90.31.10:4529 , piece_id=C3EDANWA3D3PLXDJQN5KYUHEVYVWAUE2OMBX33UII3UFTA3DOPRQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.806282081 (394h29m8.689695701s) [noise=true , address=77.90.31.13:4637 , piece_id=34WA2VCHAQUVZXH7RISTGWMTBUYDQ5YBHGQSVXJQXEZF7EXXLUIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814098678 (394h29m8.697512311s) [noise=true , address=77.90.31.10:4524 , piece_id=6YJIRRJDCIBUTRL4YXB27SYRPI6XGKWYKUAWNCGCON7BDHSIBTXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:08.814485406 (394h29m8.69789902s) [noise=true , address=77.90.31.18:4809 , piece_id=BIRC5T5KEYC2B6G73OZDDKCSUCXNZQ6UMKNHTJOOXQ5BFJW3OTIQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:10.066359268 (394h29m9.949772895s) [noise=true , address=156.146.43.225:4441 , piece_id=J6LFPF7F4RLBZRCESSUL4C7I7KWSWXGEC6YCHABJ63YP2LOJNBIA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:12.201742629 (394h29m12.085156256s) [noise=true , address=79.127.220.98:4413 , piece_id=NA7XBGOZCFXP3YF2LKCKA77FBT55LUZ6W6X4UQYYZLL45BEIS3XA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:13.472536969 (394h29m13.355950596s) [noise=true , address=45.91.116.6:4997 , piece_id=DKHOFAVZ3WXCOCUJSIQ7FSTN4Y56XZUN7MZABYPGKSBREERKMBFA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:16.917628152 (394h29m16.801041778s) [noise=true , address=45.91.116.9:4266 , piece_id=OZY7YEVACKF2HZIIOMF5MERWLJ63RQFSUVXPZAMZDXIDP5EKEZHQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:17.483834714 (394h29m17.367248345s) [noise=true , address=104.36.82.13:4192 , piece_id=5NTMTBKHCGITRBPDZKOS2T45ZZPVL77D4VLR4BQNQZA57AH4FIPQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:25.809751434 (394h29m25.693165062s) [noise=true , address=104.36.82.17:4337 , piece_id=SRZAQHNA2ULRQBJNOUC5N4VNHE2QXKCLLK4FLFH72LLXRZD3RH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:40:59.183773427 (394h29m59.067187039s) [noise=true , address=104.36.82.44:4883 , piece_id=C5NO3GCKQ4XQKPY54N2BJRNIEPC6527FSQKR6MB6XF5WBBCBVDYQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028555873 (394h30m3.911969493s) [noise=true , address=104.36.82.19:4426 , piece_id=JU7DFVYHFVGUHHD4ATGUSEMCGN5CTEJNQCZ7L5NRCMTR7JCUEBXQ, seg_pos={0 0}, upload=315464]: Done uploading piece. err: <nil> Jul 9 19:41:04.028586121 (394h30m3.911999747s) [seg_pos={0 0}, upload=315464]: Segment reached optimal threshold of 14 pieces. Jul 9 19:41:04.028700106 (394h30m3.912113722s) [noise=true , address=104.36.82.5:4864 , piece_id=MQFCBTUC4VPHQN7UQXCOKY2WRT2WKTNGNPNHTFKOJIWIH5JXQJ7A, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028743454 (394h30m3.91215708s) [noise=true , address=104.36.82.36:4642 , piece_id=AXEMFG7WBJ57Z5WPC5RTFNDOITUDNIKX3UVIVUOTB3VBX63EJ4MA, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 19:41:04.028831009 (394h30m3.912244626s) [noise=true , address=104.36.82.21:4493 , piece_id=BFUQHSHN6ZVT33BYZAYTGD6SYYZLUG77CEHDDAIYDJBB7NOWAH3Q, seg_pos={0 0}, upload=315464]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD): protocol: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58658->104.36.82.21:4493: use of closed network connection Jul 9 19:41:04.028844681 (394h30m3.912258299s) [seg_pos={0 0}, upload=315464]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 19:41:04.028918760 (394h30m3.912332377s) [upload=315464]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 19:41:04.028923227 (394h30m3.912336845s) [upload=315464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 19:41:04.186917131 (394h30m4.070330747s) [upload=315464]: Done uploading segments... Jul 9 19:41:04.186938057 (394h30m4.070351676s) [upload=315464]: Upload finished. err: <nil> Duration: 31.458075619s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)