Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 186144 reports (Page 3580 of 3723)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............X............................] duration: 36.958209137s FAIL (2025-07-09 02:43:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 36.95823682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X........X...........................X.X.......] duration: 2.435424604s FAIL (2025-07-09 02:42:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 2.435484638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX] duration: 1m0.000848883s FAIL (2025-07-09 02:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000863656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X...............X...........................] duration: 25.571346088s FAIL (2025-07-09 02:41:48 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 25.571378505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X........X........X...........................X.X......] duration: 10.999158372s FAIL (2025-07-09 02:41:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 10.999196932s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.........X.................XX..X........X.....XXX.X.XXX....] duration: 2.684961889s FAIL (2025-07-09 02:41:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.684984709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........X........X...........................X.X.....] duration: 24.28742054s FAIL (2025-07-09 02:41:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 24.287463349s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.........X.................XX..X........X.....XXX.X.XXX...] duration: 22.70689956s FAIL (2025-07-09 02:41:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.70691282s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [X.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XX] duration: 1m0.00104025s FAIL (2025-07-09 02:41:09 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001057687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration: 26.492924905s FAIL (2025-07-09 02:40:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 26.492952846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X........X........X...........................X.X....] duration: 2.318188669s FAIL (2025-07-09 02:40:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 2.318228944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............X.........................] duration: 32.866735109s FAIL (2025-07-09 02:40:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 32.866771681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.........X.................XX..X........X.....XXX.X.XXX..] duration: 21.046943918s FAIL (2025-07-09 02:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.046965114s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.X] duration: 1m0.00110819s FAIL (2025-07-09 02:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001185492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X........X........X...........................X.X...] duration: 24.668190244s FAIL (2025-07-09 02:40:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 24.668220376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............X........................] duration: 33.668423326s FAIL (2025-07-09 02:39:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 33.66847401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X........X........X...........................X.X..] duration: 2.428490635s FAIL (2025-07-09 02:39:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 2.428524724s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.........X.................XX..X........X.....XXX.X.XXX.] duration: 3.121173729s FAIL (2025-07-09 02:39:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.121312401s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.........X.................XX..X........X.....XXX.X.XXX] duration: 30.001206024s FAIL (2025-07-09 02:38:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001233927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X........X........X...........................X.X.] duration: 2.323552846s FAIL (2025-07-09 02:38:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 2.323654491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.] duration: 51.591766266s FAIL (2025-07-09 02:38:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.591784633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............X.......................] duration: 12.980781799s FAIL (2025-07-09 02:38:34 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 12.980879029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............X......................] duration: 31.18316737s FAIL (2025-07-09 02:38:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 31.183215363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X........X........X...........................X.X] duration: 30.11657925s FAIL (2025-07-09 02:38:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== s) [upload=301814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments... Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil> Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments... Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment... Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment. Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece... Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments... Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil> Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments... Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 30.116611368s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.........X.................XX..X........X.....XXX.X.XX] duration: 30.005015124s FAIL (2025-07-09 02:37:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.005051743s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X] duration: 1m0.001550876s FAIL (2025-07-09 02:37:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001620257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............X.....................] duration: 48.086698947s FAIL (2025-07-09 02:37:25 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 48.086734006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X.........X.................XX..X........X.....XXX.X.X] duration: 30.001486173s FAIL (2025-07-09 02:37:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001512761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X........X........X...........................X.] duration: 2.31083739s FAIL (2025-07-09 02:36:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== optimal: 13 errs: <nil> Jul 9 02:35:45.618851936 (377h24m45.502265562s) [upload=301805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:45.618859614 (377h24m45.50227324s) [upload=301805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:45.758241340 (377h24m45.641654966s) [upload=301805]: Done uploading segments... Jul 9 02:35:45.758260954 (377h24m45.641674574s) [upload=301805]: Upload finished. err: <nil> Jul 9 02:36:01.348279515 (377h25m1.231693154s) [upload=301809]: Starting upload Jul 9 02:36:01.348374720 (377h25m1.231788467s) [upload=301809]: Done starting upload Jul 9 02:36:01.348468898 (377h25m1.231882526s) [upload=301809]: Uploading segments... Jul 9 02:36:01.348896089 (377h25m1.232309706s) [upload=301809]: Got next segment. Inline: false Jul 9 02:36:01.348999517 (377h25m1.232413153s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:36:01.349007294 (377h25m1.232420912s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:36:01.459897633 (377h25m1.343311261s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:36:01.459994274 (377h25m1.343407908s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:36:01.460035881 (377h25m1.343449518s) [noise=true , address=104.36.82.8:4971 , piece_id=ZBKF6GIIGEYDLXC2I3X3WZKXOL22LCIZCNFVDEFUYEES3T2VK5IQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460065720 (377h25m1.343479361s) [noise=true , address=104.36.82.8:4969 , piece_id=43LLGPW5FQ2NONAWWQOAK6JIXMTOWI4ET26KIB5NHFKDFNJXJM6Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460819768 (377h25m1.344233408s) [noise=true , address=77.90.31.17:4770 , piece_id=7ORGRAI24SPG4K5B6I3PSN25ZSTY5POY2IBLIQP6FI5NQWZIZAUA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460853747 (377h25m1.344267387s) [noise=true , address=45.91.116.14:4191 , piece_id=ZG2A2BTDJWLBJCLDCR7S3RXVW7IPX33L5QJBZNVSNAIXP2SGBCIA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461325558 (377h25m1.344739194s) [noise=true , address=45.91.116.11:4118 , piece_id=HDCDLKAYAHJ466BKYOVZJDE7MWT5XDO2QNNUE7TPEKDG34SMWJNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461346666 (377h25m1.344760314s) [noise=true , address=45.91.116.5:4299 , piece_id=4ZAO2OKLU2TKBGNLUY7E4I5IZIHZL3K2TTPQ7LS46RTBIHTEZC5A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461798154 (377h25m1.345211791s) [noise=true , address=45.91.116.10:4083 , piece_id=2X7I4G5ABGQ55TLSNDKW5U6GFVU2JWYFZWE6OTPIGVWN3QGCWXPA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461843091 (377h25m1.345256723s) [noise=true , address=104.36.82.10:4077 , piece_id=5IJRBBJO2V5NJKVC2MH5MAT7QNLNXP7KIIMSFKOVWNESVB6IR2GA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462294661 (377h25m1.345708297s) [noise=true , address=45.91.116.10:4081 , piece_id=4JUPEDTBJ7NQAPZ3QI5QSHBRFGSQCEEJMO6AULZFMCBUFSHGS6JA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462345649 (377h25m1.345759352s) [noise=true , address=45.91.116.12:4226 , piece_id=6AP6KIQRP2MCGOWMH2NTLDZ7FZ47FZILYEWXRRFA4G5BVRE7LLAA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462727692 (377h25m1.346141328s) [noise=true , address=45.91.116.4:4939 , piece_id=QRXDD52ZTIOXXFMY23L5CEIDNRVQIBHFWTIT7D5UZ6RFM77MZGSQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462852105 (377h25m1.346265743s) [noise=true , address=77.90.31.11:4555 , piece_id=YEWE5LX6YZT5M7UME64XUNZAOB5BOS7H4YSG7Q2OT5OVFKSK6ODQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463134445 (377h25m1.346548115s) [noise=true , address=104.36.82.2:4731 , piece_id=RGJ42X3QSESXQ5KMIFMNV7GBXCR46JOBPK6OQWQCIFQRPNNXJZOQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463400454 (377h25m1.346814093s) [noise=true , address=104.36.82.9:4006 , piece_id=K52TWFC2PRXYY5S5FURJ37OXRKGI3QUGKZPHYFAFG2GXSDEQP5QQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463548829 (377h25m1.346962468s) [noise=true , address=104.36.82.4:4803 , piece_id=M66U7TAB3VVWABB5XLHCIKWURILVRR4LS3OKND54DO2LM53IX2JQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.464044327 (377h25m1.347457971s) [noise=true , address=104.36.82.13:4187 , piece_id=3ND2RNTYDH4X2DYGQWLSZTEJYROODPMZLISWAPQCIEQT3O3CB2AQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:02.116178521 (377h25m1.999592159s) [upload=301809]: Next returned nil segment Jul 9 02:36:02.116203219 (377h25m1.999616859s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:36:02.246208179 (377h25m2.129621794s) [noise=true , address=45.91.116.5:4299 , piece_id=4ZAO2OKLU2TKBGNLUY7E4I5IZIHZL3K2TTPQ7LS46RTBIHTEZC5A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.257147681 (377h25m2.140561303s) [noise=true , address=45.91.116.11:4118 , piece_id=HDCDLKAYAHJ466BKYOVZJDE7MWT5XDO2QNNUE7TPEKDG34SMWJNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.261581199 (377h25m2.144994831s) [noise=true , address=45.91.116.10:4081 , piece_id=4JUPEDTBJ7NQAPZ3QI5QSHBRFGSQCEEJMO6AULZFMCBUFSHGS6JA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.262914313 (377h25m2.146327932s) [noise=true , address=45.91.116.4:4939 , piece_id=QRXDD52ZTIOXXFMY23L5CEIDNRVQIBHFWTIT7D5UZ6RFM77MZGSQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.263708831 (377h25m2.147122453s) [noise=true , address=45.91.116.10:4083 , piece_id=2X7I4G5ABGQ55TLSNDKW5U6GFVU2JWYFZWE6OTPIGVWN3QGCWXPA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.263749858 (377h25m2.147163487s) [noise=true , address=104.36.82.9:4006 , piece_id=K52TWFC2PRXYY5S5FURJ37OXRKGI3QUGKZPHYFAFG2GXSDEQP5QQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.264509387 (377h25m2.147923003s) [noise=true , address=45.91.116.12:4226 , piece_id=6AP6KIQRP2MCGOWMH2NTLDZ7FZ47FZILYEWXRRFA4G5BVRE7LLAA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.264775529 (377h25m2.148189143s) [noise=true , address=45.91.116.14:4191 , piece_id=ZG2A2BTDJWLBJCLDCR7S3RXVW7IPX33L5QJBZNVSNAIXP2SGBCIA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.270458653 (377h25m2.153872276s) [noise=true , address=77.90.31.17:4770 , piece_id=7ORGRAI24SPG4K5B6I3PSN25ZSTY5POY2IBLIQP6FI5NQWZIZAUA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.274115051 (377h25m2.157528697s) [noise=true , address=77.90.31.11:4555 , piece_id=YEWE5LX6YZT5M7UME64XUNZAOB5BOS7H4YSG7Q2OT5OVFKSK6ODQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:13.364764124 (377h25m13.24817774s) [noise=true , address=104.36.82.13:4187 , piece_id=3ND2RNTYDH4X2DYGQWLSZTEJYROODPMZLISWAPQCIEQT3O3CB2AQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:28.989370105 (377h25m28.872783722s) [noise=true , address=104.36.82.2:4731 , piece_id=RGJ42X3QSESXQ5KMIFMNV7GBXCR46JOBPK6OQWQCIFQRPNNXJZOQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:31.077335690 (377h25m30.960749317s) [noise=true , address=104.36.82.10:4077 , piece_id=5IJRBBJO2V5NJKVC2MH5MAT7QNLNXP7KIIMSFKOVWNESVB6IR2GA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection; write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection Jul 9 02:36:31.077373358 (377h25m30.960786977s) [noise=true , address=104.36.82.4:4803 , piece_id=M66U7TAB3VVWABB5XLHCIKWURILVRR4LS3OKND54DO2LM53IX2JQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:36:31.077436665 (377h25m30.960850284s) [noise=true , address=104.36.82.8:4969 , piece_id=43LLGPW5FQ2NONAWWQOAK6JIXMTOWI4ET26KIB5NHFKDFNJXJM6Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection Jul 9 02:36:31.077455291 (377h25m30.960868913s) [noise=true , address=104.36.82.8:4971 , piece_id=ZBKF6GIIGEYDLXC2I3X3WZKXOL22LCIZCNFVDEFUYEES3T2VK5IQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection Jul 9 02:36:31.077473370 (377h25m30.960887001s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:36:31.181973226 (377h25m31.065386846s) [upload=301809]: Done uploading segments... Jul 9 02:36:31.182001480 (377h25m31.065415107s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.310877206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..] duration: 8.315444086s FAIL (2025-07-09 02:36:36 UTC) orbiter warning: context deadline exceeded Duration: 8.315465615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X........X........X...........................X] duration: 30.106400853s FAIL (2025-07-09 02:36:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== optimal: 13 errs: <nil> Jul 9 02:35:45.618851936 (377h24m45.502265562s) [upload=301805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:45.618859614 (377h24m45.50227324s) [upload=301805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:45.758241340 (377h24m45.641654966s) [upload=301805]: Done uploading segments... Jul 9 02:35:45.758260954 (377h24m45.641674574s) [upload=301805]: Upload finished. err: <nil> Jul 9 02:36:01.348279515 (377h25m1.231693154s) [upload=301809]: Starting upload Jul 9 02:36:01.348374720 (377h25m1.231788467s) [upload=301809]: Done starting upload Jul 9 02:36:01.348468898 (377h25m1.231882526s) [upload=301809]: Uploading segments... Jul 9 02:36:01.348896089 (377h25m1.232309706s) [upload=301809]: Got next segment. Inline: false Jul 9 02:36:01.348999517 (377h25m1.232413153s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:36:01.349007294 (377h25m1.232420912s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:36:01.459897633 (377h25m1.343311261s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:36:01.459994274 (377h25m1.343407908s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:36:01.460035881 (377h25m1.343449518s) [noise=true , address=104.36.82.8:4971 , piece_id=ZBKF6GIIGEYDLXC2I3X3WZKXOL22LCIZCNFVDEFUYEES3T2VK5IQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460065720 (377h25m1.343479361s) [noise=true , address=104.36.82.8:4969 , piece_id=43LLGPW5FQ2NONAWWQOAK6JIXMTOWI4ET26KIB5NHFKDFNJXJM6Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460819768 (377h25m1.344233408s) [noise=true , address=77.90.31.17:4770 , piece_id=7ORGRAI24SPG4K5B6I3PSN25ZSTY5POY2IBLIQP6FI5NQWZIZAUA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.460853747 (377h25m1.344267387s) [noise=true , address=45.91.116.14:4191 , piece_id=ZG2A2BTDJWLBJCLDCR7S3RXVW7IPX33L5QJBZNVSNAIXP2SGBCIA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461325558 (377h25m1.344739194s) [noise=true , address=45.91.116.11:4118 , piece_id=HDCDLKAYAHJ466BKYOVZJDE7MWT5XDO2QNNUE7TPEKDG34SMWJNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461346666 (377h25m1.344760314s) [noise=true , address=45.91.116.5:4299 , piece_id=4ZAO2OKLU2TKBGNLUY7E4I5IZIHZL3K2TTPQ7LS46RTBIHTEZC5A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461798154 (377h25m1.345211791s) [noise=true , address=45.91.116.10:4083 , piece_id=2X7I4G5ABGQ55TLSNDKW5U6GFVU2JWYFZWE6OTPIGVWN3QGCWXPA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.461843091 (377h25m1.345256723s) [noise=true , address=104.36.82.10:4077 , piece_id=5IJRBBJO2V5NJKVC2MH5MAT7QNLNXP7KIIMSFKOVWNESVB6IR2GA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462294661 (377h25m1.345708297s) [noise=true , address=45.91.116.10:4081 , piece_id=4JUPEDTBJ7NQAPZ3QI5QSHBRFGSQCEEJMO6AULZFMCBUFSHGS6JA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462345649 (377h25m1.345759352s) [noise=true , address=45.91.116.12:4226 , piece_id=6AP6KIQRP2MCGOWMH2NTLDZ7FZ47FZILYEWXRRFA4G5BVRE7LLAA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462727692 (377h25m1.346141328s) [noise=true , address=45.91.116.4:4939 , piece_id=QRXDD52ZTIOXXFMY23L5CEIDNRVQIBHFWTIT7D5UZ6RFM77MZGSQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.462852105 (377h25m1.346265743s) [noise=true , address=77.90.31.11:4555 , piece_id=YEWE5LX6YZT5M7UME64XUNZAOB5BOS7H4YSG7Q2OT5OVFKSK6ODQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463134445 (377h25m1.346548115s) [noise=true , address=104.36.82.2:4731 , piece_id=RGJ42X3QSESXQ5KMIFMNV7GBXCR46JOBPK6OQWQCIFQRPNNXJZOQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463400454 (377h25m1.346814093s) [noise=true , address=104.36.82.9:4006 , piece_id=K52TWFC2PRXYY5S5FURJ37OXRKGI3QUGKZPHYFAFG2GXSDEQP5QQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.463548829 (377h25m1.346962468s) [noise=true , address=104.36.82.4:4803 , piece_id=M66U7TAB3VVWABB5XLHCIKWURILVRR4LS3OKND54DO2LM53IX2JQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:01.464044327 (377h25m1.347457971s) [noise=true , address=104.36.82.13:4187 , piece_id=3ND2RNTYDH4X2DYGQWLSZTEJYROODPMZLISWAPQCIEQT3O3CB2AQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:36:02.116178521 (377h25m1.999592159s) [upload=301809]: Next returned nil segment Jul 9 02:36:02.116203219 (377h25m1.999616859s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:36:02.246208179 (377h25m2.129621794s) [noise=true , address=45.91.116.5:4299 , piece_id=4ZAO2OKLU2TKBGNLUY7E4I5IZIHZL3K2TTPQ7LS46RTBIHTEZC5A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.257147681 (377h25m2.140561303s) [noise=true , address=45.91.116.11:4118 , piece_id=HDCDLKAYAHJ466BKYOVZJDE7MWT5XDO2QNNUE7TPEKDG34SMWJNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.261581199 (377h25m2.144994831s) [noise=true , address=45.91.116.10:4081 , piece_id=4JUPEDTBJ7NQAPZ3QI5QSHBRFGSQCEEJMO6AULZFMCBUFSHGS6JA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.262914313 (377h25m2.146327932s) [noise=true , address=45.91.116.4:4939 , piece_id=QRXDD52ZTIOXXFMY23L5CEIDNRVQIBHFWTIT7D5UZ6RFM77MZGSQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.263708831 (377h25m2.147122453s) [noise=true , address=45.91.116.10:4083 , piece_id=2X7I4G5ABGQ55TLSNDKW5U6GFVU2JWYFZWE6OTPIGVWN3QGCWXPA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.263749858 (377h25m2.147163487s) [noise=true , address=104.36.82.9:4006 , piece_id=K52TWFC2PRXYY5S5FURJ37OXRKGI3QUGKZPHYFAFG2GXSDEQP5QQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.264509387 (377h25m2.147923003s) [noise=true , address=45.91.116.12:4226 , piece_id=6AP6KIQRP2MCGOWMH2NTLDZ7FZ47FZILYEWXRRFA4G5BVRE7LLAA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.264775529 (377h25m2.148189143s) [noise=true , address=45.91.116.14:4191 , piece_id=ZG2A2BTDJWLBJCLDCR7S3RXVW7IPX33L5QJBZNVSNAIXP2SGBCIA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.270458653 (377h25m2.153872276s) [noise=true , address=77.90.31.17:4770 , piece_id=7ORGRAI24SPG4K5B6I3PSN25ZSTY5POY2IBLIQP6FI5NQWZIZAUA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:02.274115051 (377h25m2.157528697s) [noise=true , address=77.90.31.11:4555 , piece_id=YEWE5LX6YZT5M7UME64XUNZAOB5BOS7H4YSG7Q2OT5OVFKSK6ODQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:13.364764124 (377h25m13.24817774s) [noise=true , address=104.36.82.13:4187 , piece_id=3ND2RNTYDH4X2DYGQWLSZTEJYROODPMZLISWAPQCIEQT3O3CB2AQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:28.989370105 (377h25m28.872783722s) [noise=true , address=104.36.82.2:4731 , piece_id=RGJ42X3QSESXQ5KMIFMNV7GBXCR46JOBPK6OQWQCIFQRPNNXJZOQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:36:31.077335690 (377h25m30.960749317s) [noise=true , address=104.36.82.10:4077 , piece_id=5IJRBBJO2V5NJKVC2MH5MAT7QNLNXP7KIIMSFKOVWNESVB6IR2GA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection; write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49388->104.36.82.10:4077: use of closed network connection Jul 9 02:36:31.077373358 (377h25m30.960786977s) [noise=true , address=104.36.82.4:4803 , piece_id=M66U7TAB3VVWABB5XLHCIKWURILVRR4LS3OKND54DO2LM53IX2JQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:36:31.077436665 (377h25m30.960850284s) [noise=true , address=104.36.82.8:4969 , piece_id=43LLGPW5FQ2NONAWWQOAK6JIXMTOWI4ET26KIB5NHFKDFNJXJM6Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47500->104.36.82.8:4969: use of closed network connection Jul 9 02:36:31.077455291 (377h25m30.960868913s) [noise=true , address=104.36.82.8:4971 , piece_id=ZBKF6GIIGEYDLXC2I3X3WZKXOL22LCIZCNFVDEFUYEES3T2VK5IQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49966->104.36.82.8:4971: use of closed network connection Jul 9 02:36:31.077473370 (377h25m30.960887001s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:36:31.181973226 (377h25m31.065386846s) [upload=301809]: Done uploading segments... Jul 9 02:36:31.182001480 (377h25m31.065415107s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.106430849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX.] duration: 39.411618737s FAIL (2025-07-09 02:36:28 UTC) orbiter warning: context deadline exceeded Duration: 39.411650417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............X....................] duration: 46.201044625s FAIL (2025-07-09 02:36:14 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 46.201081791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.....X.........X.................XX..X........X.....XXX.X.] duration: 24.406470692s FAIL (2025-07-09 02:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.406498302s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [X.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX] duration: 1m0.001164752s FAIL (2025-07-09 02:35:48 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001183616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X........X........X...........................] duration: 2.427884602s FAIL (2025-07-09 02:35:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.427922712s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X.....X.........X.................XX..X........X.....XXX.X] duration: 30.00203023s FAIL (2025-07-09 02:35:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002063294s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X........X........X..........................] duration: 25.849284673s FAIL (2025-07-09 02:34:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 25.84932909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............X...................] duration: 37.088649321s FAIL (2025-07-09 02:34:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 37.088682653s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXX] duration: 1m0.001249575s FAIL (2025-07-09 02:34:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001276912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............X..................] duration: 42.219212494s FAIL (2025-07-09 02:34:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 42.219244983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.....X.........X.................XX..X........X.....XXX.] duration: 8.033985909s FAIL (2025-07-09 02:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.034011426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X........X........X.........................] duration: 2.291089228s FAIL (2025-07-09 02:33:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.29113316s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XX] duration: 1m0.000381351s FAIL (2025-07-09 02:33:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00040227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.....X.........X.................XX..X........X.....XXX] duration: 30.000607634s FAIL (2025-07-09 02:33:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000634559s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............X.................] duration: 44.148505776s FAIL (2025-07-09 02:32:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 44.148543326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X........X........X........................] duration: 2.303075144s FAIL (2025-07-09 02:32:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.303102954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.X] duration: 1m0.001672957s FAIL (2025-07-09 02:32:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001706053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.X.....X.........X.................XX..X........X.....XX] duration: 30.001339283s FAIL (2025-07-09 02:32:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001358153s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............X................] duration: 34.081664688s FAIL (2025-07-09 02:32:11 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 34.081696129s core uplink us-select-1 us1-placement-20

1
...
3578
3579
3580
3581
3582
...
3723