Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 186144 reports (Page 3581 of 3723)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X.....X.........X.................XX..X........X.....X] duration: 30.002151371s FAIL (2025-07-09 02:31:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002174345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X........X........X.......................] duration: 2.250039445s FAIL (2025-07-09 02:31:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.250078849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...............X...............] duration: 26.265387464s FAIL (2025-07-09 02:31:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 26.265411362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.] duration: 47.127611357s FAIL (2025-07-09 02:31:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.127629459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.X.....X.........X.................XX..X........X.....] duration: 8.896092359s FAIL (2025-07-09 02:31:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.896123792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...............X..............] duration: 32.334371079s FAIL (2025-07-09 02:31:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 32.334399542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X........X........X......................] duration: 15.300171221s FAIL (2025-07-09 02:31:04 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 15.300341754s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.X.....X.........X.................XX..X........X....] duration: 21.008781618s FAIL (2025-07-09 02:30:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.008794283s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X......X.X.....X.........X.................XX..X........X...] duration: 14.959084244s FAIL (2025-07-09 02:30:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.959104753s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X] duration: 1m0.001020312s FAIL (2025-07-09 02:30:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001046638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...............X.............] duration: 32.374578235s FAIL (2025-07-09 02:30:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 32.374608312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X........X........X.....................] duration: 17.179809346s FAIL (2025-07-09 02:29:58 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 17.179921954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...............X............] duration: 39.567895151s FAIL (2025-07-09 02:29:14 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 39.567939713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.] duration: 40.655299529s FAIL (2025-07-09 02:29:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.655313249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......X.X.....X.........X.................XX..X........X..] duration: 3.437985957s FAIL (2025-07-09 02:29:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438001897s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X......X.X.....X.........X.................XX..X........X.] duration: 22.340643096s FAIL (2025-07-09 02:28:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.340672316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X........X........X....................] duration: 2.321628583s FAIL (2025-07-09 02:28:52 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.321657894s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X......X.X.....X.........X.................XX..X........X] duration: 30.00104528s FAIL (2025-07-09 02:28:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001064942s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [X..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX] duration: 1m0.002091284s FAIL (2025-07-09 02:28:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002121709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X........X........X...................] duration: 2.478359559s FAIL (2025-07-09 02:28:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.478395187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X...............X...........] duration: 24.509571452s FAIL (2025-07-09 02:27:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 24.509598723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X........X........X..................] duration: 2.390522197s FAIL (2025-07-09 02:27:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.390554094s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.X] duration: 1m0.001900089s FAIL (2025-07-09 02:27:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001932589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...............X..........] duration: 27.631282292s FAIL (2025-07-09 02:27:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 27.631308708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X......X.X.....X.........X.................XX..X........] duration: 2.879462637s FAIL (2025-07-09 02:26:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.879490824s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...............X.........] duration: 33.103290842s FAIL (2025-07-09 02:26:53 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 33.103313865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X........X........X.................] duration: 2.275915356s FAIL (2025-07-09 02:26:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.275950118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X........X........X................] duration: 2.359956714s FAIL (2025-07-09 02:26:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.359999803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...............X........] duration: 56.799111261s FAIL (2025-07-09 02:26:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 56.79913773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X.X.....X.........X.................XX..X.......] duration: 16.186633463s FAIL (2025-07-09 02:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.186668385s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.] duration: 46.035881933s FAIL (2025-07-09 02:26:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.035908234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........................X........X........X...............] duration: 2.671178915s FAIL (2025-07-09 02:26:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.67121142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........................X........X........X..............] duration: 18.777089782s FAIL (2025-07-09 02:25:25 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 18.777193183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X......X.X.....X.........X.................XX..X......] duration: 6.852267853s FAIL (2025-07-09 02:25:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.852297511s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX] duration: 1m0.00169484s FAIL (2025-07-09 02:25:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001740376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X......X.X.....X.........X.................XX..X.....] duration: 21.218269029s FAIL (2025-07-09 02:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.218308423s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........................X........X........X.............] duration: 2.827986892s FAIL (2025-07-09 02:24:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.828020375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...............X.......] duration: 2.564757023s FAIL (2025-07-09 02:24:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 2.564779838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...............X......] duration: 47.10578903s FAIL (2025-07-09 02:24:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 47.105823661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.X] duration: 1m0.000783073s FAIL (2025-07-09 02:24:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000857046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........................X........X........X............] duration: 2.373046624s FAIL (2025-07-09 02:24:08 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.37307772s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X......X.X.....X.........X.................XX..X....] duration: 27.171549498s FAIL (2025-07-09 02:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.171579086s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........................X........X........X...........] duration: 2.322220455s FAIL (2025-07-09 02:23:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.322241557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........................X........X........X..........] duration: 2.298778113s FAIL (2025-07-09 02:23:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.298807995s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X......X.X.....X.........X.................XX..X...] duration: 3.326057636s FAIL (2025-07-09 02:23:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.32609389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...............X.....] duration: 10.651254646s FAIL (2025-07-09 02:22:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 10.651302047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...............X....] duration: 37.232717788s FAIL (2025-07-09 02:22:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 37.232737148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.] duration: 21.346070514s FAIL (2025-07-09 02:22:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.346090004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.........................X........X........X.........] duration: 2.867728693s FAIL (2025-07-09 02:22:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.867810312s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [X.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX] duration: 1m0.000660186s FAIL (2025-07-09 02:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000682491s core uplink us-select-1 us1-placement-20

1
...
3579
3580
3581
3582
3583
...
3723