Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 347557 reports (Page 6435 of 6952)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........................X.XX...XX..X..XXXXX....XX.XX..X.....] duration: 24.731680561s FAIL (2025-07-26 23:32:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.731711339s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [X..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X] duration: 1m0.108502642s FAIL (2025-07-26 23:32:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 23:31:00.367172428 (59h55m31.557471001s) [upload=57459]: Got next segment. Inline: false Jul 26 23:31:00.367230418 (59h55m31.557528989s) [upload=57459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:00.367243316 (59h55m31.557541901s) [upload=57459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:00.575320973 (59h55m31.765619543s) [seg_pos={0 0}, upload=57459]: Begin upload segment... Jul 26 23:31:00.575371635 (59h55m31.765670204s) [seg_pos={0 0}, upload=57459]: Done begin upload segment. Jul 26 23:31:00.575411467 (59h55m31.765710049s) [noise=true , address=45.91.116.14:4194 , piece_id=R2FOSL6RDLKHZWAEBHCDSZUGCIBKYGLUN36U7XPAKIR2TIIOOXDQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.575803035 (59h55m31.766101614s) [noise=true , address=104.36.82.54:4175 , piece_id=CLONQ6LUJWG3SXZ2F7BPSDIZWP35HVB4IVDU4EGPUWGQ5MWGM4EA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.576028392 (59h55m31.766326973s) [noise=true , address=104.36.82.53:4130 , piece_id=A2VK3YGU52G55SIEFEFHWSN6WVV5GFA6DVX75NUHNKPHNL7VZDAQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.576809066 (59h55m31.767107656s) [noise=true , address=156.146.43.225:4448 , piece_id=N62EYLM75UMWB2QRJAHARDQBBIHQRDEBRQUJAV3QAC2BUQ7IZIKA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.577171685 (59h55m31.767470272s) [noise=true , address=79.127.201.215:4345 , piece_id=TFJQ5EHX2EU54JMXMF2NC7MXZ5WDPK5V5W63AANRQZ7KTF7ZVL6Q, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.577805190 (59h55m31.768103774s) [noise=true , address=64.49.13.137:4502 , piece_id=CO5473DAC7K7CFJG32OBV54XRP7RJOPKGGLYAFTAOX2WBHPAJONA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578202717 (59h55m31.768501307s) [noise=true , address=77.90.31.10:4531 , piece_id=Y4T5SQTHHCKA7TXU3RIAOBR35V6Q6VMNYOGSDYG6IVFVSE7PZW5A, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578262207 (59h55m31.768560792s) [noise=true , address=79.127.205.233:4650 , piece_id=3YDZ4BT2JSGFMV6AI33YWWODC2FAEGBRP6NIWMI2RINO5QYDWZFA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578615161 (59h55m31.768913743s) [noise=true , address=79.127.205.246:4283 , piece_id=CUBNPFJA2XMQPKN3LPNQXIJ7J6TR7GFS7CIRI22AYH7L6A6T2F7Q, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578632568 (59h55m31.768931152s) [noise=true , address=77.90.31.12:4615 , piece_id=QQIDNK5XZDE2JK3NBNWHAJ43XVUSCIWWGYCB4AB2EKG3N3ULOJBQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578944365 (59h55m31.769242955s) [noise=true , address=79.127.220.101:4041 , piece_id=IIB63L55VZLRI5JN2JO5BZG7MSIHDH225CYCG4234CDN6I77BRVQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.578962843 (59h55m31.769261433s) [noise=true , address=104.36.82.46:4499 , piece_id=SZ2CPGMPM5VAI4QPVFMBW2HX74UKMSXGLYFN2WDGGQS77EIAN6SQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.579289481 (59h55m31.769588062s) [noise=true , address=79.127.205.235:4679 , piece_id=3IBXDFDI3HFHQTJHUA6ZRUF7MERBLO7UTOMAVZWFSAV7B6HL4WTA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.579330033 (59h55m31.76962862s) [noise=true , address=79.127.205.245:4243 , piece_id=LTCCQ5YO3XIFHEMQ7NANZW4MBPKXLYKKQYIYMCSTUNDXUGM7OUUQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.579608540 (59h55m31.769907127s) [noise=true , address=79.127.205.234:4608 , piece_id=IWYRYHGCPXOHF22FJCOLZHIXVNPXGDFHHML6ANPLBGN2NTF4LEEA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.579730067 (59h55m31.770028658s) [noise=true , address=104.36.82.55:4202 , piece_id=GGYDCIKOOPUOUWFRINH4DZMJZSTNKQZB3PXER2DQJARSXDJ44PFA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.580756568 (59h55m31.771055149s) [noise=true , address=104.36.82.47:4539 , piece_id=2NTXOGAJ7AM3UQQ73TCI6PTTNC6PQJMDFPRRPT2EU3RFRTYDJYQQ, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:00.581664560 (59h55m31.771963145s) [noise=true , address=79.127.201.214:4321 , piece_id=XGADPHGWNLTAG7RLXQSDMRGPGZ5FJIGZVFC6CWHAQBWOIOKJ2GNA, seg_pos={0 0}, upload=57459]: Uploading piece... Jul 26 23:31:01.147112596 (59h55m32.337411176s) [upload=57459]: Next returned nil segment Jul 26 23:31:01.147135993 (59h55m32.337434581s) [upload=57459]: Waiting for error group managing segments... Jul 26 23:31:01.150396195 (59h55m32.340694767s) [noise=true , address=79.127.220.101:4041 , piece_id=IIB63L55VZLRI5JN2JO5BZG7MSIHDH225CYCG4234CDN6I77BRVQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:01.159899435 (59h55m32.350198018s) [noise=true , address=156.146.43.225:4448 , piece_id=N62EYLM75UMWB2QRJAHARDQBBIHQRDEBRQUJAV3QAC2BUQ7IZIKA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:01.251832119 (59h55m32.442130699s) [noise=true , address=79.127.201.214:4321 , piece_id=XGADPHGWNLTAG7RLXQSDMRGPGZ5FJIGZVFC6CWHAQBWOIOKJ2GNA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:01.331031042 (59h55m32.521329619s) [noise=true , address=45.91.116.14:4194 , piece_id=R2FOSL6RDLKHZWAEBHCDSZUGCIBKYGLUN36U7XPAKIR2TIIOOXDQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.342985223 (59h55m32.533283794s) [noise=true , address=77.90.31.10:4531 , piece_id=Y4T5SQTHHCKA7TXU3RIAOBR35V6Q6VMNYOGSDYG6IVFVSE7PZW5A, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.849208366 (59h55m33.039506952s) [noise=true , address=79.127.205.233:4650 , piece_id=3YDZ4BT2JSGFMV6AI33YWWODC2FAEGBRP6NIWMI2RINO5QYDWZFA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.853345538 (59h55m33.043644109s) [noise=true , address=79.127.205.246:4283 , piece_id=CUBNPFJA2XMQPKN3LPNQXIJ7J6TR7GFS7CIRI22AYH7L6A6T2F7Q, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.925738832 (59h55m33.116037417s) [noise=true , address=79.127.205.234:4608 , piece_id=IWYRYHGCPXOHF22FJCOLZHIXVNPXGDFHHML6ANPLBGN2NTF4LEEA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.932610638 (59h55m33.122909228s) [noise=true , address=79.127.205.245:4243 , piece_id=LTCCQ5YO3XIFHEMQ7NANZW4MBPKXLYKKQYIYMCSTUNDXUGM7OUUQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:01.958576208 (59h55m33.148874791s) [noise=true , address=79.127.205.235:4679 , piece_id=3IBXDFDI3HFHQTJHUA6ZRUF7MERBLO7UTOMAVZWFSAV7B6HL4WTA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:04.614468060 (59h55m35.804766639s) [noise=true , address=64.49.13.137:4502 , piece_id=CO5473DAC7K7CFJG32OBV54XRP7RJOPKGGLYAFTAOX2WBHPAJONA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:15.170109312 (59h55m46.360407877s) [noise=true , address=104.36.82.53:4130 , piece_id=A2VK3YGU52G55SIEFEFHWSN6WVV5GFA6DVX75NUHNKPHNL7VZDAQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:18.086612071 (59h55m49.27691066s) [noise=true , address=104.36.82.55:4202 , piece_id=GGYDCIKOOPUOUWFRINH4DZMJZSTNKQZB3PXER2DQJARSXDJ44PFA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:29.662054583 (59h56m0.852353155s) [noise=true , address=104.36.82.54:4175 , piece_id=CLONQ6LUJWG3SXZ2F7BPSDIZWP35HVB4IVDU4EGPUWGQ5MWGM4EA, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:36.745034341 (59h56m7.93533291s) [noise=true , address=104.36.82.47:4539 , piece_id=2NTXOGAJ7AM3UQQ73TCI6PTTNC6PQJMDFPRRPT2EU3RFRTYDJYQQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:31:45.307571679 (59h56m16.497870249s) [noise=true , address=79.127.201.215:4345 , piece_id=TFJQ5EHX2EU54JMXMF2NC7MXZ5WDPK5V5W63AANRQZ7KTF7ZVL6Q, seg_pos={0 0}, upload=57459]: Done uploading piece. err: <nil> Jul 26 23:32:00.091210174 (59h56m31.281508769s) [noise=true , address=77.90.31.12:4615 , piece_id=QQIDNK5XZDE2JK3NBNWHAJ43XVUSCIWWGYCB4AB2EKG3N3ULOJBQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: rpc: dial tcp 77.90.31.12:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:00.091248016 (59h56m31.281546599s) [noise=true , address=104.36.82.46:4499 , piece_id=SZ2CPGMPM5VAI4QPVFMBW2HX74UKMSXGLYFN2WDGGQS77EIAN6SQ, seg_pos={0 0}, upload=57459]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 139.59.144.22:36958->104.36.82.46:4499: use of closed network connection; write tcp 139.59.144.22:36958->104.36.82.46:4499: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36958->104.36.82.46:4499: use of closed network connection Jul 26 23:32:00.091304038 (59h56m31.281602619s) [seg_pos={0 0}, upload=57459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:32:00.196962319 (59h56m31.387260889s) [upload=57459]: Done uploading segments... Jul 26 23:32:00.196997519 (59h56m31.387296087s) [upload=57459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.108531414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [.X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.107997182s FAIL (2025-07-26 23:31:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.108023989s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.......X.......X........XX....X....X.X..X........XX.] duration: 17.314408003s FAIL (2025-07-26 23:31:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.314426876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [....................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 30.003292323s FAIL (2025-07-26 23:31:34 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== FpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection Jul 26 23:30:47.602447620 (59h55m18.792746202s) [seg_pos={0 0}, upload=57448]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:30:47.707507832 (59h55m18.897806419s) [upload=57448]: Done uploading segments... Jul 26 23:30:47.707542109 (59h55m18.897840677s) [upload=57448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:31:04.685577145 (59h55m35.875875736s) [upload=57462]: Starting upload Jul 26 23:31:04.685666521 (59h55m35.875965109s) [upload=57462]: Done starting upload Jul 26 23:31:04.685774274 (59h55m35.876072843s) [upload=57462]: Uploading segments... Jul 26 23:31:04.686283845 (59h55m35.876582437s) [upload=57462]: Got next segment. Inline: false Jul 26 23:31:04.686426491 (59h55m35.876725073s) [upload=57462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:04.686435471 (59h55m35.87673405s) [upload=57462]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:04.884394340 (59h55m36.074692929s) [seg_pos={0 0}, upload=57462]: Begin upload segment... Jul 26 23:31:04.884618539 (59h55m36.074917109s) [seg_pos={0 0}, upload=57462]: Done begin upload segment. Jul 26 23:31:04.884669154 (59h55m36.07496774s) [noise=true , address=104.36.82.11:4113 , piece_id=I4MDLDM5ZSM45KEP5H3T6NTSML44P4KZIB2AVZDKNDHOVFOZVBKA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.884680800 (59h55m36.074979385s) [noise=true , address=104.36.82.5:4837 , piece_id=HGOYFDOZ4OSPDCCVQH2UCSWNVGBEWGUJQPHKCUK5UJ4AKLW7BKQA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.885158197 (59h55m36.075456782s) [noise=true , address=77.90.31.14:4662 , piece_id=MUUTXXMXJJDOWRV3GQH4TMLMNQJ2K5E767THPLSUDK3LI7K35EZA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.885196883 (59h55m36.075495452s) [noise=true , address=77.90.31.14:4663 , piece_id=NQ4GDMGTICFQ4R5BDQM5H3X5LB2HFDYJJVT2GGYSVOOGRBGBINOA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.885603894 (59h55m36.075902476s) [noise=true , address=77.90.31.15:4698 , piece_id=NAXHYGVXBIE5VOSCGCRSJ6W5BUMHOMIZDH2C5YV6NTSDXOBKJ65A, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.885622054 (59h55m36.075920644s) [noise=true , address=104.36.82.15:4259 , piece_id=XNZTJK7G5G6UT4EEUBBRZL7T6B7IALOSRZILBPXITTXNZOATDDHA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886057150 (59h55m36.076355741s) [noise=true , address=45.91.116.7:4010 , piece_id=AF6M5IZB6PVPL4TFRFU4EFW33GMDKXQXUTIGHWHGJBKVBTE534XA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886079546 (59h55m36.076378131s) [noise=true , address=104.36.82.14:4221 , piece_id=PZIWCAU2ZY2PUWN4KTMITEXAL3PPHCAR7RTPZUG67Q7GKEH6OXWA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886381431 (59h55m36.076680015s) [noise=true , address=77.90.31.21:4878 , piece_id=V3YNCOUA3P5LDGSCST4F5ZX3BANS4P2KVATMK5YOBLRGXJEF5LDQ, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886444204 (59h55m36.076742794s) [noise=true , address=45.91.116.5:4299 , piece_id=PO54S2DGFG5OW5QJMYTH3KCPXZQ4JOALEWLEYX3AV46FM6H7W5MA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886656066 (59h55m36.076954632s) [noise=true , address=45.91.116.13:4155 , piece_id=WGQESYUKA7P7GFS2P4LFOAHLQSZNO4VNKBXXRHNRKSDOP24WOJYQ, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886807138 (59h55m36.077105729s) [noise=true , address=45.91.116.8:4048 , piece_id=GSFSXLXMSDHHMOVH5IW6N7BNFU4HTGDM3EOUUD77ZXYTKKDZ6OGA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.886927045 (59h55m36.077225625s) [noise=true , address=45.91.116.7:4009 , piece_id=H6LD3G2XURPDNOTQTQMNIPA6KPZGPYYLCKLBPDH5WZZSKY6OGR2Q, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.887187086 (59h55m36.077485673s) [noise=true , address=77.90.31.12:4592 , piece_id=CFH4EM7TRMAYLT325YZYTUZY5OYD4J42L2TLRK2JQAEYRIWLPQHQ, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.889658050 (59h55m36.07995664s) [noise=true , address=104.36.82.2:4731 , piece_id=PNNG2CEJZQ3FYG2L7MLDR7KPUPYO2XVYPLGSJYVNFYQEONESWDVQ, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:04.891691700 (59h55m36.081990286s) [noise=true , address=45.91.116.10:4082 , piece_id=7O5FI3X5LRK3KU4NDHSCCFIZKAMQIUMSYEILUSKXXMBFK7UTKLAA, seg_pos={0 0}, upload=57462]: Uploading piece... Jul 26 23:31:05.515399785 (59h55m36.705698374s) [upload=57462]: Next returned nil segment Jul 26 23:31:05.515426316 (59h55m36.705724905s) [upload=57462]: Waiting for error group managing segments... Jul 26 23:31:05.642859878 (59h55m36.833158446s) [noise=true , address=45.91.116.10:4082 , piece_id=7O5FI3X5LRK3KU4NDHSCCFIZKAMQIUMSYEILUSKXXMBFK7UTKLAA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.644850345 (59h55m36.835148925s) [noise=true , address=77.90.31.21:4878 , piece_id=V3YNCOUA3P5LDGSCST4F5ZX3BANS4P2KVATMK5YOBLRGXJEF5LDQ, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.647077430 (59h55m36.837375996s) [noise=true , address=45.91.116.7:4009 , piece_id=H6LD3G2XURPDNOTQTQMNIPA6KPZGPYYLCKLBPDH5WZZSKY6OGR2Q, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.647966512 (59h55m36.838265098s) [noise=true , address=45.91.116.13:4155 , piece_id=WGQESYUKA7P7GFS2P4LFOAHLQSZNO4VNKBXXRHNRKSDOP24WOJYQ, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.649141768 (59h55m36.839440334s) [noise=true , address=77.90.31.15:4698 , piece_id=NAXHYGVXBIE5VOSCGCRSJ6W5BUMHOMIZDH2C5YV6NTSDXOBKJ65A, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.649992868 (59h55m36.840291449s) [noise=true , address=77.90.31.14:4663 , piece_id=NQ4GDMGTICFQ4R5BDQM5H3X5LB2HFDYJJVT2GGYSVOOGRBGBINOA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.650724993 (59h55m36.841023562s) [noise=true , address=77.90.31.14:4662 , piece_id=MUUTXXMXJJDOWRV3GQH4TMLMNQJ2K5E767THPLSUDK3LI7K35EZA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.652974225 (59h55m36.843272812s) [noise=true , address=45.91.116.8:4048 , piece_id=GSFSXLXMSDHHMOVH5IW6N7BNFU4HTGDM3EOUUD77ZXYTKKDZ6OGA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.654075277 (59h55m36.844373864s) [noise=true , address=45.91.116.7:4010 , piece_id=AF6M5IZB6PVPL4TFRFU4EFW33GMDKXQXUTIGHWHGJBKVBTE534XA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:05.719760660 (59h55m36.910059231s) [noise=true , address=45.91.116.5:4299 , piece_id=PO54S2DGFG5OW5QJMYTH3KCPXZQ4JOALEWLEYX3AV46FM6H7W5MA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:25.521263725 (59h55m56.711562294s) [noise=true , address=104.36.82.11:4113 , piece_id=I4MDLDM5ZSM45KEP5H3T6NTSML44P4KZIB2AVZDKNDHOVFOZVBKA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:29.456005723 (59h56m0.646304308s) [noise=true , address=104.36.82.15:4259 , piece_id=XNZTJK7G5G6UT4EEUBBRZL7T6B7IALOSRZILBPXITTXNZOATDDHA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:29.464804558 (59h56m0.655103129s) [noise=true , address=104.36.82.2:4731 , piece_id=PNNG2CEJZQ3FYG2L7MLDR7KPUPYO2XVYPLGSJYVNFYQEONESWDVQ, seg_pos={0 0}, upload=57462]: Done uploading piece. err: <nil> Jul 26 23:31:29.464834126 (59h56m0.6551327s) [seg_pos={0 0}, upload=57462]: Segment reached optimal threshold of 13 pieces. Jul 26 23:31:29.465037492 (59h56m0.655336072s) [noise=true , address=104.36.82.5:4837 , piece_id=HGOYFDOZ4OSPDCCVQH2UCSWNVGBEWGUJQPHKCUK5UJ4AKLW7BKQA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Jul 26 23:31:29.465577061 (59h56m0.65587564s) [noise=true , address=77.90.31.12:4592 , piece_id=CFH4EM7TRMAYLT325YZYTUZY5OYD4J42L2TLRK2JQAEYRIWLPQHQ, seg_pos={0 0}, upload=57462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 139.59.144.22:39136->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:29.466513676 (59h56m0.656812255s) [noise=true , address=104.36.82.14:4221 , piece_id=PZIWCAU2ZY2PUWN4KTMITEXAL3PPHCAR7RTPZUG67Q7GKEH6OXWA, seg_pos={0 0}, upload=57462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 139.59.144.22:35010->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35010->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35010->104.36.82.14:4221: use of closed network connection Jul 26 23:31:29.466529620 (59h56m0.656828186s) [seg_pos={0 0}, upload=57462]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:31:29.466604854 (59h56m0.656903448s) [upload=57462]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:31:29.466611142 (59h56m0.65690973s) [upload=57462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:31:29.609697257 (59h56m0.799995837s) [upload=57462]: Done uploading segments... Jul 26 23:31:29.609717491 (59h56m0.800016064s) [upload=57462]: Upload finished. err: <nil> Duration: 30.003325823s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......X.X.......X.......X........XX....X....X.X..X........XX] duration: 1m0.000612662s FAIL (2025-07-26 23:31:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000627149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........................X.XX...XX..X..XXXXX....XX.XX..X....] duration: 2.717636394s FAIL (2025-07-26 23:31:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717664563s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [XX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.] duration: 58.101650449s FAIL (2025-07-26 23:30:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 7s) [upload=57432]: Uploading segments... Jul 26 23:29:01.790719647 (59h53m32.981018236s) [upload=57432]: Got next segment. Inline: false Jul 26 23:29:01.790775173 (59h53m32.981073758s) [upload=57432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:01.790781973 (59h53m32.981080543s) [upload=57432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:01.987960934 (59h53m33.178259523s) [seg_pos={0 0}, upload=57432]: Begin upload segment... Jul 26 23:29:01.988029561 (59h53m33.17832815s) [seg_pos={0 0}, upload=57432]: Done begin upload segment. Jul 26 23:29:01.988043141 (59h53m33.178341734s) [noise=true , address=79.127.220.101:4041 , piece_id=ZVXBNP2W22W4SGLDTNRAXWYMRHJ2RD23UCCWFK7IWF5BJDG4RCIQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.988056663 (59h53m33.178355255s) [noise=true , address=79.127.201.215:4356 , piece_id=JS4PXM6TDHAYEBCIZMYK7XXVYUQI6ZPTPJ4GZYQ4CKZQ6QYXGTXA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.988842908 (59h53m33.179141585s) [noise=true , address=156.146.43.225:4448 , piece_id=WF3BLEQK5PEKXWBSLOOYGZIPD5XVXXHC5PKFRTQR7UGQNN4M6VUQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.989326262 (59h53m33.179624849s) [noise=true , address=104.36.82.52:4096 , piece_id=4WMWSOMD5QI3CZJ42PXLQ7MHFZUHWQMLRWWOG2QHIJG4EVUGEUWA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.989737568 (59h53m33.180036161s) [noise=true , address=104.36.82.54:4181 , piece_id=3KFMGKO4HP46EEOM4DNTJCEGZEM5JBVCHIFVSUMXVD5R4CPLESVQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990256174 (59h53m33.180554783s) [noise=true , address=79.127.205.235:4691 , piece_id=TPOA5JJQ77ESJE6EEEIWWHMGWMDYKZEYUWBPWYJZTY4TBFNFHZIA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990712248 (59h53m33.181010841s) [noise=true , address=77.90.31.15:4726 , piece_id=DCCHPOA4PCJKBSJOKPQIOZ3RBOKLQNFFOM7JJ6DKO5LCWAQUJR4Q, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990740799 (59h53m33.181039392s) [noise=true , address=45.91.116.11:4121 , piece_id=QMJ2XRXGV7TLNTRUKOZ4O2XPSS2GBTGVW3ARNKEYX2VO2JHM4XWA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991115774 (59h53m33.181414368s) [noise=true , address=45.91.116.14:4206 , piece_id=BU4KIBXPBAXTR4IVZOBY6TNUCZ7E4UHSW7YDSG5B7CMGSTFY32SA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991512158 (59h53m33.181810752s) [noise=true , address=104.36.82.46:4485 , piece_id=W6SWFXRHMFPM5ZXKJ2FRCUEJEBVLLPLWP54JX5MGA3FGGOHXMLIQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991523560 (59h53m33.181822152s) [noise=true , address=64.49.13.138:4482 , piece_id=GJQKA4XAWSKO2NAQ3V43RSIJ6YC23IWTCY7P6V46M7IJMCDO3LTQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991906159 (59h53m33.182204755s) [noise=true , address=77.90.31.21:4896 , piece_id=MHQALYW2IN5HV5A6JU6Z4ZQ2PCOWHGJ7QYHJXL3DJ4FEOIZHPSAA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.992346471 (59h53m33.182645061s) [noise=true , address=79.127.219.38:4018 , piece_id=NZ2OWPF2RVSPYBKYMRLOP3URMP43QPNZG4VIBT5YJWOZZMPOMHFQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.994068170 (59h53m33.184366787s) [noise=true , address=104.36.82.56:4246 , piece_id=3JJB4TJZ5ATOK5JX2LDPM3WQ53JTNHIFAV255JHVASY7SQWNEUOQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.994541705 (59h53m33.184840299s) [noise=true , address=104.36.82.16:4303 , piece_id=JV4A3YRIO5VJOWSUZECPRS76EFCRR5BKJXKIGPZFCILZTEC2SVSQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.995023028 (59h53m33.185321635s) [noise=true , address=79.127.205.234:4615 , piece_id=Y4VWVWX4E24PGT7XRSWEYN3XAGF73Z44CHVVN2RBEZNFIBWJ5QAA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.996045235 (59h53m33.18634385s) [noise=true , address=77.90.31.13:4651 , piece_id=WRVMZBD6MWRLJ7BLGL5W6WPXTADZAEFTJQR3H5WXRAVXMAGDZDMQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.996850185 (59h53m33.18714877s) [noise=true , address=104.36.82.47:4551 , piece_id=MO4J3IR3KXGI6BPZ6TLHMOX3ZLWNDL3NIK7NGSZM652DGVDOENVQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:02.447041459 (59h53m33.637340047s) [noise=true , address=79.127.220.101:4041 , piece_id=ZVXBNP2W22W4SGLDTNRAXWYMRHJ2RD23UCCWFK7IWF5BJDG4RCIQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 26 23:29:02.513814100 (59h53m33.704112689s) [noise=true , address=156.146.43.225:4448 , piece_id=WF3BLEQK5PEKXWBSLOOYGZIPD5XVXXHC5PKFRTQR7UGQNN4M6VUQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:29:02.559098204 (59h53m33.749396795s) [upload=57432]: Next returned nil segment Jul 26 23:29:02.559124312 (59h53m33.7494229s) [upload=57432]: Waiting for error group managing segments... Jul 26 23:29:02.696303520 (59h53m33.886602104s) [noise=true , address=77.90.31.21:4896 , piece_id=MHQALYW2IN5HV5A6JU6Z4ZQ2PCOWHGJ7QYHJXL3DJ4FEOIZHPSAA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:02.855627256 (59h53m34.045925835s) [noise=true , address=45.91.116.11:4121 , piece_id=QMJ2XRXGV7TLNTRUKOZ4O2XPSS2GBTGVW3ARNKEYX2VO2JHM4XWA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.229399815 (59h53m34.41969838s) [noise=true , address=79.127.219.38:4018 , piece_id=NZ2OWPF2RVSPYBKYMRLOP3URMP43QPNZG4VIBT5YJWOZZMPOMHFQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.401894167 (59h53m34.592192757s) [noise=true , address=79.127.205.235:4691 , piece_id=TPOA5JJQ77ESJE6EEEIWWHMGWMDYKZEYUWBPWYJZTY4TBFNFHZIA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.479892786 (59h53m34.670191359s) [noise=true , address=79.127.205.234:4615 , piece_id=Y4VWVWX4E24PGT7XRSWEYN3XAGF73Z44CHVVN2RBEZNFIBWJ5QAA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:07.649950149 (59h53m38.84024874s) [noise=true , address=64.49.13.138:4482 , piece_id=GJQKA4XAWSKO2NAQ3V43RSIJ6YC23IWTCY7P6V46M7IJMCDO3LTQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:08.492248748 (59h53m39.682547332s) [noise=true , address=45.91.116.14:4206 , piece_id=BU4KIBXPBAXTR4IVZOBY6TNUCZ7E4UHSW7YDSG5B7CMGSTFY32SA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:09.166329195 (59h53m40.356627774s) [noise=true , address=79.127.201.215:4356 , piece_id=JS4PXM6TDHAYEBCIZMYK7XXVYUQI6ZPTPJ4GZYQ4CKZQ6QYXGTXA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:24.205514540 (59h53m55.395813106s) [noise=true , address=104.36.82.54:4181 , piece_id=3KFMGKO4HP46EEOM4DNTJCEGZEM5JBVCHIFVSUMXVD5R4CPLESVQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:32.926420598 (59h54m4.116719164s) [noise=true , address=104.36.82.56:4246 , piece_id=3JJB4TJZ5ATOK5JX2LDPM3WQ53JTNHIFAV255JHVASY7SQWNEUOQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:36.534683589 (59h54m7.724982162s) [noise=true , address=77.90.31.13:4651 , piece_id=WRVMZBD6MWRLJ7BLGL5W6WPXTADZAEFTJQR3H5WXRAVXMAGDZDMQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:57.402874678 (59h54m28.593173267s) [noise=true , address=104.36.82.46:4485 , piece_id=W6SWFXRHMFPM5ZXKJ2FRCUEJEBVLLPLWP54JX5MGA3FGGOHXMLIQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:30:01.517749535 (59h54m32.708048119s) [noise=true , address=104.36.82.47:4551 , piece_id=MO4J3IR3KXGI6BPZ6TLHMOX3ZLWNDL3NIK7NGSZM652DGVDOENVQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518130559 (59h54m32.708429152s) [noise=true , address=77.90.31.15:4726 , piece_id=DCCHPOA4PCJKBSJOKPQIOZ3RBOKLQNFFOM7JJ6DKO5LCWAQUJR4Q, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518398932 (59h54m32.708697513s) [noise=true , address=104.36.82.52:4096 , piece_id=4WMWSOMD5QI3CZJ42PXLQ7MHFZUHWQMLRWWOG2QHIJG4EVUGEUWA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518639062 (59h54m32.708937643s) [noise=true , address=104.36.82.16:4303 , piece_id=JV4A3YRIO5VJOWSUZECPRS76EFCRR5BKJXKIGPZFCILZTEC2SVSQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection; write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection Jul 26 23:30:01.518706085 (59h54m32.709004678s) [seg_pos={0 0}, upload=57432]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:30:01.661162039 (59h54m32.85146061s) [upload=57432]: Done uploading segments... Jul 26 23:30:01.661222438 (59h54m32.851521019s) [upload=57432]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 58.101677252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........................X.XX...XX..X..XXXXX....XX.XX..X...] duration: 10.901938232s FAIL (2025-07-26 23:30:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.90219714s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [..X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.106822359s FAIL (2025-07-26 23:30:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.106853508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.....................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 30.107459282s FAIL (2025-07-26 23:30:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== {0 0}, upload=57448]: Done begin upload segment. Jul 26 23:30:17.985447113 (59h54m49.175745705s) [noise=true , address=45.91.116.6:4975 , piece_id=NRDWGRRXVF2OJDAF3NMI6AUMA2P6V2NMT7E247YGKZCJPCLKZUTA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.985462504 (59h54m49.175761074s) [noise=true , address=104.36.82.16:4293 , piece_id=4BWRMIVIJVL45HB7HSOXCSYEOJLZUSMHYTGVCTEGUNLPYTMDS7VQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.985905663 (59h54m49.176204245s) [noise=true , address=104.36.82.4:4801 , piece_id=P2ZOVEWIMCUXASE4QPNFD5OIO7IJWV5WCZVZ7YZISIOLMG2JVBRQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.986275621 (59h54m49.176574204s) [noise=true , address=45.91.116.8:4045 , piece_id=6B6H2FEVS45ZE5RBEA7JRABIWFD4ZFBZRNGEVD45MACYARGAKLSA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.986433569 (59h54m49.176732152s) [noise=true , address=104.36.82.13:4186 , piece_id=WLNCA4EYAK32IL2I53TXW5JRV43PLIMRAE3BRJVGCLQ5ATRF32IQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.986603593 (59h54m49.176902176s) [noise=true , address=45.91.116.5:4299 , piece_id=XYGYMY3UZJ76ZT7SOFEFLAB66XATMMDMH6EOACGH3R73PSLXQSUQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.986781154 (59h54m49.177079741s) [noise=true , address=77.90.31.12:4592 , piece_id=G6FR4GIMLLALPT2YVIUNHME2SDBZEMPDY7DBPJTS5GDXA5FIVYFA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.987128767 (59h54m49.177427357s) [noise=true , address=104.36.82.14:4221 , piece_id=5C5SVWXJGBYJDZSSFCVGYICTHRAQW2EGYJNNI6DFMVTHAN3W3SRQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.987446852 (59h54m49.177745438s) [noise=true , address=77.90.31.10:4517 , piece_id=FSLWN24LGZGWRZEQDLHPCB2MLKT2IVYMTOFI5SWTXZSJYTFVURLA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.987739720 (59h54m49.178038301s) [noise=true , address=104.36.82.10:4079 , piece_id=S7UDFYEHRUCS3G65ZA5JI23CGBUEPYK2FTSDXMMIHZXEWC53QQLA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.987855896 (59h54m49.178154478s) [noise=true , address=104.36.82.5:4837 , piece_id=NXC6DXMN2M6JA2QLX42DY5WNQFUN4I6AQBGJUFLXSI7L5FYHP4KQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.988199515 (59h54m49.178498097s) [noise=true , address=104.36.82.9:4005 , piece_id=BWL47X4PSXUIFA5K6YHF5KOJNVA42HX5ME2K7XX6XPUS6NBYR7KA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.988644621 (59h54m49.17894319s) [noise=true , address=45.91.116.9:4263 , piece_id=UP7QS44E3JKGJY3OXAJ65KGWLNO5PARR2JTIXKI5BREENRCLNEBQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.988915467 (59h54m49.179214055s) [noise=true , address=104.36.82.15:4259 , piece_id=GLGBUDBTRJUNKYFGR536D2Z3HEGKIXATWCMVC6RNWPZIS5LHKY7Q, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.989313733 (59h54m49.179612315s) [noise=true , address=77.90.31.19:4842 , piece_id=C6JV3E3WBXJFBXEPRFQWYVKIMJUZRXODCEG4ST5NXU33DRNJD3IQ, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:17.989530809 (59h54m49.179829394s) [noise=true , address=104.36.82.2:4731 , piece_id=QVCK2HVS5YLY6EBCQY6BTKYQVSRCGU2PIAXIVPBS4WFSFBFAY2AA, seg_pos={0 0}, upload=57448]: Uploading piece... Jul 26 23:30:18.594342568 (59h54m49.784641148s) [upload=57448]: Next returned nil segment Jul 26 23:30:18.594357666 (59h54m49.784656246s) [upload=57448]: Waiting for error group managing segments... Jul 26 23:30:18.728141280 (59h54m49.918439849s) [noise=true , address=45.91.116.6:4975 , piece_id=NRDWGRRXVF2OJDAF3NMI6AUMA2P6V2NMT7E247YGKZCJPCLKZUTA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:18.728188935 (59h54m49.918487507s) [noise=true , address=77.90.31.10:4517 , piece_id=FSLWN24LGZGWRZEQDLHPCB2MLKT2IVYMTOFI5SWTXZSJYTFVURLA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:18.769955559 (59h54m49.960254141s) [noise=true , address=45.91.116.8:4045 , piece_id=6B6H2FEVS45ZE5RBEA7JRABIWFD4ZFBZRNGEVD45MACYARGAKLSA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:19.018322820 (59h54m50.208621394s) [noise=true , address=45.91.116.5:4299 , piece_id=XYGYMY3UZJ76ZT7SOFEFLAB66XATMMDMH6EOACGH3R73PSLXQSUQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:19.221393276 (59h54m50.411691849s) [noise=true , address=45.91.116.9:4263 , piece_id=UP7QS44E3JKGJY3OXAJ65KGWLNO5PARR2JTIXKI5BREENRCLNEBQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:20.785352049 (59h54m51.975650622s) [noise=true , address=77.90.31.19:4842 , piece_id=C6JV3E3WBXJFBXEPRFQWYVKIMJUZRXODCEG4ST5NXU33DRNJD3IQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:39.513733205 (59h55m10.70403177s) [noise=true , address=104.36.82.4:4801 , piece_id=P2ZOVEWIMCUXASE4QPNFD5OIO7IJWV5WCZVZ7YZISIOLMG2JVBRQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:46.588695846 (59h55m17.778994435s) [noise=true , address=104.36.82.15:4259 , piece_id=GLGBUDBTRJUNKYFGR536D2Z3HEGKIXATWCMVC6RNWPZIS5LHKY7Q, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:47.209385771 (59h55m18.399684352s) [noise=true , address=104.36.82.9:4005 , piece_id=BWL47X4PSXUIFA5K6YHF5KOJNVA42HX5ME2K7XX6XPUS6NBYR7KA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: <nil> Jul 26 23:30:47.602041094 (59h55m18.792339683s) [noise=true , address=104.36.82.13:4186 , piece_id=WLNCA4EYAK32IL2I53TXW5JRV43PLIMRAE3BRJVGCLQ5ATRF32IQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:44656->104.36.82.13:4186: use of closed network connection; write tcp 139.59.144.22:44656->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44656->104.36.82.13:4186: use of closed network connection Jul 26 23:30:47.602110014 (59h55m18.792408596s) [noise=true , address=104.36.82.5:4837 , piece_id=NXC6DXMN2M6JA2QLX42DY5WNQFUN4I6AQBGJUFLXSI7L5FYHP4KQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58130->104.36.82.5:4837: use of closed network connection; write tcp 139.59.144.22:58130->104.36.82.5:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58130->104.36.82.5:4837: use of closed network connection Jul 26 23:30:47.602187593 (59h55m18.792486175s) [noise=true , address=104.36.82.14:4221 , piece_id=5C5SVWXJGBYJDZSSFCVGYICTHRAQW2EGYJNNI6DFMVTHAN3W3SRQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:34614->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:34614->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34614->104.36.82.14:4221: use of closed network connection Jul 26 23:30:47.602280106 (59h55m18.792578686s) [noise=true , address=77.90.31.12:4592 , piece_id=G6FR4GIMLLALPT2YVIUNHME2SDBZEMPDY7DBPJTS5GDXA5FIVYFA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:47.602355037 (59h55m18.792653617s) [noise=true , address=104.36.82.10:4079 , piece_id=S7UDFYEHRUCS3G65ZA5JI23CGBUEPYK2FTSDXMMIHZXEWC53QQLA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:49348->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:49348->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49348->104.36.82.10:4079: use of closed network connection Jul 26 23:30:47.602386592 (59h55m18.792685179s) [noise=true , address=104.36.82.16:4293 , piece_id=4BWRMIVIJVL45HB7HSOXCSYEOJLZUSMHYTGVCTEGUNLPYTMDS7VQ, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:57820->104.36.82.16:4293: use of closed network connection; write tcp 139.59.144.22:57820->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57820->104.36.82.16:4293: use of closed network connection Jul 26 23:30:47.602415104 (59h55m18.792713674s) [noise=true , address=104.36.82.2:4731 , piece_id=QVCK2HVS5YLY6EBCQY6BTKYQVSRCGU2PIAXIVPBS4WFSFBFAY2AA, seg_pos={0 0}, upload=57448]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51970->104.36.82.2:4731: use of closed network connection Jul 26 23:30:47.602447620 (59h55m18.792746202s) [seg_pos={0 0}, upload=57448]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:30:47.707507832 (59h55m18.897806419s) [upload=57448]: Done uploading segments... Jul 26 23:30:47.707542109 (59h55m18.897840677s) [upload=57448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.107484757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...........................X.XX...XX..X..XXXXX....XX.XX..X..] duration: 3.298272983s FAIL (2025-07-26 23:30:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.298300417s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X......X.X.......X.......X........XX....X....X.X..X........X] duration: 1m0.001409123s FAIL (2025-07-26 23:30:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001430693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [......................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.127482895s FAIL (2025-07-26 23:30:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 26 23:29:07.700785347 (59h53m38.891083926s) [upload=57423]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:29:34.447496239 (59h54m5.637794841s) [upload=57437]: Starting upload Jul 26 23:29:34.447591282 (59h54m5.637889871s) [upload=57437]: Done starting upload Jul 26 23:29:34.447687991 (59h54m5.637986564s) [upload=57437]: Uploading segments... Jul 26 23:29:34.447925548 (59h54m5.638224128s) [upload=57437]: Got next segment. Inline: false Jul 26 23:29:34.447964705 (59h54m5.638263284s) [upload=57437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:34.447971258 (59h54m5.638269827s) [upload=57437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:34.558620365 (59h54m5.748918944s) [seg_pos={0 0}, upload=57437]: Begin upload segment... Jul 26 23:29:34.558699083 (59h54m5.748997669s) [noise=true , address=45.91.116.9:4261 , piece_id=7CVQRHDRB2YKL4RK5FO3JD6GK3MHDJEHTUJP5PFFQS3IOG5U377A, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.559109201 (59h54m5.749407781s) [noise=true , address=77.90.31.14:4661 , piece_id=R2UOUMYUO43ZTK5KTEWX6PPXHLK3VRIV2JJL7LITMAYMZCS4SUOA, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.559508463 (59h54m5.749807048s) [noise=true , address=104.36.82.13:4185 , piece_id=BOE5DEV7N2D2NKCTGSGYDGNKUELLL4LLH4BDFIBZKXVU6G72J56Q, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.559882162 (59h54m5.750180744s) [noise=true , address=104.36.82.9:4007 , piece_id=BUYYMH72MX5T4DXRW5AZMN33KGTKQF436JZBV6BKGHSCVQM6OH2Q, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.559904445 (59h54m5.750203076s) [seg_pos={0 0}, upload=57437]: Done begin upload segment. Jul 26 23:29:34.559957121 (59h54m5.750255704s) [noise=true , address=104.36.82.6:4876 , piece_id=JZFP4KFSJXAGZ2WGRF7GXMMD2XYX5VSHDX3CJ4TQFIS67J4NEEXQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.560360995 (59h54m5.750659612s) [noise=true , address=45.91.116.6:4974 , piece_id=OLCI4R7EAJF2EV5VRC6CIHPSHL46ENDDNHUHE2PTF7CJTMSB64YQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.560819849 (59h54m5.751118431s) [noise=true , address=45.91.116.14:4189 , piece_id=RFI7CFS45232D4X6ARPURLF5GGFI6SSD63A7FUHIIAI3IVJEU6VA, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.561141567 (59h54m5.751440145s) [noise=true , address=104.36.82.7:4911 , piece_id=MMINI6Q2ST6CVVOGLTI4S2RZKTTRPNEYHCR3GJMXKOYOHM6KGSOA, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.561426429 (59h54m5.751725011s) [noise=true , address=77.90.31.19:4842 , piece_id=L3PLBAXFG7LJBJO3BXHULRDOVZ62IRRWTEL4JLBOV5EDJASMGCIQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.561826319 (59h54m5.752124899s) [noise=true , address=45.91.116.13:4154 , piece_id=CWTWHOV5KOB6JZ56KRQAZDDSREXSWKF4U5KEDJRVHI7UVWF5NQ2A, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.562153041 (59h54m5.752451621s) [noise=true , address=104.36.82.8:4969 , piece_id=Q3YW3QDI4Y4DO7NJA5KDQ6XOGLGKLW6FQPTICETD53MXACEBNB3Q, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.563008121 (59h54m5.75330671s) [noise=true , address=104.36.82.10:4077 , piece_id=RAQZTM4ZUHYDJQKP2HNJGKUWH7DMDIF75OSGKYKUCTWLGTW75HOQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.565143802 (59h54m5.755442387s) [noise=true , address=45.91.116.12:4225 , piece_id=EB75GA7HYLJKID3KBLLJIG772GWD7QMH6KZRP65GE4LVJSDEUSLQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.565575960 (59h54m5.75587454s) [noise=true , address=104.36.82.3:4767 , piece_id=IANOQ5PODNIZEKVR457QBJDQWMHKAM2Q3SWJYBFLC6FPX6444RVA, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.566021028 (59h54m5.756319613s) [noise=true , address=45.91.116.4:4937 , piece_id=ZQNK56DVX5PS335AAZ5U4PC5HSVQRHB55V724RE2JTSQJ7CUFZKQ, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:34.566847788 (59h54m5.757146368s) [noise=true , address=104.36.82.5:4837 , piece_id=XSEZG3467VCQOKTT3EI54HOJQHQQR5S2CIZEXCEZ6EN6GXJRX6HA, seg_pos={0 0}, upload=57437]: Uploading piece... Jul 26 23:29:35.179956824 (59h54m6.370255418s) [upload=57437]: Next returned nil segment Jul 26 23:29:35.179982616 (59h54m6.370281202s) [upload=57437]: Waiting for error group managing segments... Jul 26 23:29:35.313348899 (59h54m6.503647472s) [noise=true , address=45.91.116.6:4974 , piece_id=OLCI4R7EAJF2EV5VRC6CIHPSHL46ENDDNHUHE2PTF7CJTMSB64YQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.314204869 (59h54m6.504503449s) [noise=true , address=45.91.116.9:4261 , piece_id=7CVQRHDRB2YKL4RK5FO3JD6GK3MHDJEHTUJP5PFFQS3IOG5U377A, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.316844496 (59h54m6.50714308s) [noise=true , address=77.90.31.14:4661 , piece_id=R2UOUMYUO43ZTK5KTEWX6PPXHLK3VRIV2JJL7LITMAYMZCS4SUOA, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.317123330 (59h54m6.507421904s) [noise=true , address=45.91.116.12:4225 , piece_id=EB75GA7HYLJKID3KBLLJIG772GWD7QMH6KZRP65GE4LVJSDEUSLQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.333662542 (59h54m6.523961128s) [noise=true , address=45.91.116.13:4154 , piece_id=CWTWHOV5KOB6JZ56KRQAZDDSREXSWKF4U5KEDJRVHI7UVWF5NQ2A, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.334548998 (59h54m6.524847571s) [noise=true , address=45.91.116.4:4937 , piece_id=ZQNK56DVX5PS335AAZ5U4PC5HSVQRHB55V724RE2JTSQJ7CUFZKQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:35.429454514 (59h54m6.619753084s) [noise=true , address=45.91.116.14:4189 , piece_id=RFI7CFS45232D4X6ARPURLF5GGFI6SSD63A7FUHIIAI3IVJEU6VA, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:37.457631456 (59h54m8.647930029s) [noise=true , address=77.90.31.19:4842 , piece_id=L3PLBAXFG7LJBJO3BXHULRDOVZ62IRRWTEL4JLBOV5EDJASMGCIQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:29:55.706384306 (59h54m26.896682892s) [noise=true , address=104.36.82.9:4007 , piece_id=BUYYMH72MX5T4DXRW5AZMN33KGTKQF436JZBV6BKGHSCVQM6OH2Q, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:30:03.374169789 (59h54m34.564468358s) [noise=true , address=104.36.82.6:4876 , piece_id=JZFP4KFSJXAGZ2WGRF7GXMMD2XYX5VSHDX3CJ4TQFIS67J4NEEXQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:30:03.917035852 (59h54m35.107334424s) [noise=true , address=104.36.82.7:4911 , piece_id=MMINI6Q2ST6CVVOGLTI4S2RZKTTRPNEYHCR3GJMXKOYOHM6KGSOA, seg_pos={0 0}, upload=57437]: Done uploading piece. err: <nil> Jul 26 23:30:04.168418083 (59h54m35.358716672s) [noise=true , address=104.36.82.8:4969 , piece_id=Q3YW3QDI4Y4DO7NJA5KDQ6XOGLGKLW6FQPTICETD53MXACEBNB3Q, seg_pos={0 0}, upload=57437]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:42676->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:42676->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42676->104.36.82.8:4969: use of closed network connection Jul 26 23:30:04.168905094 (59h54m35.359203665s) [noise=true , address=104.36.82.10:4077 , piece_id=RAQZTM4ZUHYDJQKP2HNJGKUWH7DMDIF75OSGKYKUCTWLGTW75HOQ, seg_pos={0 0}, upload=57437]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 139.59.144.22:58718->104.36.82.10:4077: use of closed network connection; write tcp 139.59.144.22:58718->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58718->104.36.82.10:4077: use of closed network connection Jul 26 23:30:04.168969686 (59h54m35.359268256s) [noise=true , address=104.36.82.3:4767 , piece_id=IANOQ5PODNIZEKVR457QBJDQWMHKAM2Q3SWJYBFLC6FPX6444RVA, seg_pos={0 0}, upload=57437]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:33990->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:33990->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33990->104.36.82.3:4767: use of closed network connection Jul 26 23:30:04.169019105 (59h54m35.359317677s) [noise=true , address=104.36.82.13:4185 , piece_id=BOE5DEV7N2D2NKCTGSGYDGNKUELLL4LLH4BDFIBZKXVU6G72J56Q, seg_pos={0 0}, upload=57437]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:47462->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:47462->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47462->104.36.82.13:4185: use of closed network connection Jul 26 23:30:04.169300071 (59h54m35.359598649s) [noise=true , address=104.36.82.5:4837 , piece_id=XSEZG3467VCQOKTT3EI54HOJQHQQR5S2CIZEXCEZ6EN6GXJRX6HA, seg_pos={0 0}, upload=57437]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:04.169341982 (59h54m35.359640568s) [seg_pos={0 0}, upload=57437]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:30:04.292691091 (59h54m35.48298968s) [upload=57437]: Done uploading segments... Jul 26 23:30:04.292766751 (59h54m35.483065345s) [upload=57437]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.127534013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................X.XX...XX..X..XXXXX....XX.XX..X.] duration: 4.703866353s FAIL (2025-07-26 23:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.703908388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX] duration: 1m0.146320341s FAIL (2025-07-26 23:30:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 7s) [upload=57432]: Uploading segments... Jul 26 23:29:01.790719647 (59h53m32.981018236s) [upload=57432]: Got next segment. Inline: false Jul 26 23:29:01.790775173 (59h53m32.981073758s) [upload=57432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:01.790781973 (59h53m32.981080543s) [upload=57432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:01.987960934 (59h53m33.178259523s) [seg_pos={0 0}, upload=57432]: Begin upload segment... Jul 26 23:29:01.988029561 (59h53m33.17832815s) [seg_pos={0 0}, upload=57432]: Done begin upload segment. Jul 26 23:29:01.988043141 (59h53m33.178341734s) [noise=true , address=79.127.220.101:4041 , piece_id=ZVXBNP2W22W4SGLDTNRAXWYMRHJ2RD23UCCWFK7IWF5BJDG4RCIQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.988056663 (59h53m33.178355255s) [noise=true , address=79.127.201.215:4356 , piece_id=JS4PXM6TDHAYEBCIZMYK7XXVYUQI6ZPTPJ4GZYQ4CKZQ6QYXGTXA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.988842908 (59h53m33.179141585s) [noise=true , address=156.146.43.225:4448 , piece_id=WF3BLEQK5PEKXWBSLOOYGZIPD5XVXXHC5PKFRTQR7UGQNN4M6VUQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.989326262 (59h53m33.179624849s) [noise=true , address=104.36.82.52:4096 , piece_id=4WMWSOMD5QI3CZJ42PXLQ7MHFZUHWQMLRWWOG2QHIJG4EVUGEUWA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.989737568 (59h53m33.180036161s) [noise=true , address=104.36.82.54:4181 , piece_id=3KFMGKO4HP46EEOM4DNTJCEGZEM5JBVCHIFVSUMXVD5R4CPLESVQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990256174 (59h53m33.180554783s) [noise=true , address=79.127.205.235:4691 , piece_id=TPOA5JJQ77ESJE6EEEIWWHMGWMDYKZEYUWBPWYJZTY4TBFNFHZIA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990712248 (59h53m33.181010841s) [noise=true , address=77.90.31.15:4726 , piece_id=DCCHPOA4PCJKBSJOKPQIOZ3RBOKLQNFFOM7JJ6DKO5LCWAQUJR4Q, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.990740799 (59h53m33.181039392s) [noise=true , address=45.91.116.11:4121 , piece_id=QMJ2XRXGV7TLNTRUKOZ4O2XPSS2GBTGVW3ARNKEYX2VO2JHM4XWA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991115774 (59h53m33.181414368s) [noise=true , address=45.91.116.14:4206 , piece_id=BU4KIBXPBAXTR4IVZOBY6TNUCZ7E4UHSW7YDSG5B7CMGSTFY32SA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991512158 (59h53m33.181810752s) [noise=true , address=104.36.82.46:4485 , piece_id=W6SWFXRHMFPM5ZXKJ2FRCUEJEBVLLPLWP54JX5MGA3FGGOHXMLIQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991523560 (59h53m33.181822152s) [noise=true , address=64.49.13.138:4482 , piece_id=GJQKA4XAWSKO2NAQ3V43RSIJ6YC23IWTCY7P6V46M7IJMCDO3LTQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.991906159 (59h53m33.182204755s) [noise=true , address=77.90.31.21:4896 , piece_id=MHQALYW2IN5HV5A6JU6Z4ZQ2PCOWHGJ7QYHJXL3DJ4FEOIZHPSAA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.992346471 (59h53m33.182645061s) [noise=true , address=79.127.219.38:4018 , piece_id=NZ2OWPF2RVSPYBKYMRLOP3URMP43QPNZG4VIBT5YJWOZZMPOMHFQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.994068170 (59h53m33.184366787s) [noise=true , address=104.36.82.56:4246 , piece_id=3JJB4TJZ5ATOK5JX2LDPM3WQ53JTNHIFAV255JHVASY7SQWNEUOQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.994541705 (59h53m33.184840299s) [noise=true , address=104.36.82.16:4303 , piece_id=JV4A3YRIO5VJOWSUZECPRS76EFCRR5BKJXKIGPZFCILZTEC2SVSQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.995023028 (59h53m33.185321635s) [noise=true , address=79.127.205.234:4615 , piece_id=Y4VWVWX4E24PGT7XRSWEYN3XAGF73Z44CHVVN2RBEZNFIBWJ5QAA, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.996045235 (59h53m33.18634385s) [noise=true , address=77.90.31.13:4651 , piece_id=WRVMZBD6MWRLJ7BLGL5W6WPXTADZAEFTJQR3H5WXRAVXMAGDZDMQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:01.996850185 (59h53m33.18714877s) [noise=true , address=104.36.82.47:4551 , piece_id=MO4J3IR3KXGI6BPZ6TLHMOX3ZLWNDL3NIK7NGSZM652DGVDOENVQ, seg_pos={0 0}, upload=57432]: Uploading piece... Jul 26 23:29:02.447041459 (59h53m33.637340047s) [noise=true , address=79.127.220.101:4041 , piece_id=ZVXBNP2W22W4SGLDTNRAXWYMRHJ2RD23UCCWFK7IWF5BJDG4RCIQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 26 23:29:02.513814100 (59h53m33.704112689s) [noise=true , address=156.146.43.225:4448 , piece_id=WF3BLEQK5PEKXWBSLOOYGZIPD5XVXXHC5PKFRTQR7UGQNN4M6VUQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:29:02.559098204 (59h53m33.749396795s) [upload=57432]: Next returned nil segment Jul 26 23:29:02.559124312 (59h53m33.7494229s) [upload=57432]: Waiting for error group managing segments... Jul 26 23:29:02.696303520 (59h53m33.886602104s) [noise=true , address=77.90.31.21:4896 , piece_id=MHQALYW2IN5HV5A6JU6Z4ZQ2PCOWHGJ7QYHJXL3DJ4FEOIZHPSAA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:02.855627256 (59h53m34.045925835s) [noise=true , address=45.91.116.11:4121 , piece_id=QMJ2XRXGV7TLNTRUKOZ4O2XPSS2GBTGVW3ARNKEYX2VO2JHM4XWA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.229399815 (59h53m34.41969838s) [noise=true , address=79.127.219.38:4018 , piece_id=NZ2OWPF2RVSPYBKYMRLOP3URMP43QPNZG4VIBT5YJWOZZMPOMHFQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.401894167 (59h53m34.592192757s) [noise=true , address=79.127.205.235:4691 , piece_id=TPOA5JJQ77ESJE6EEEIWWHMGWMDYKZEYUWBPWYJZTY4TBFNFHZIA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:03.479892786 (59h53m34.670191359s) [noise=true , address=79.127.205.234:4615 , piece_id=Y4VWVWX4E24PGT7XRSWEYN3XAGF73Z44CHVVN2RBEZNFIBWJ5QAA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:07.649950149 (59h53m38.84024874s) [noise=true , address=64.49.13.138:4482 , piece_id=GJQKA4XAWSKO2NAQ3V43RSIJ6YC23IWTCY7P6V46M7IJMCDO3LTQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:08.492248748 (59h53m39.682547332s) [noise=true , address=45.91.116.14:4206 , piece_id=BU4KIBXPBAXTR4IVZOBY6TNUCZ7E4UHSW7YDSG5B7CMGSTFY32SA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:09.166329195 (59h53m40.356627774s) [noise=true , address=79.127.201.215:4356 , piece_id=JS4PXM6TDHAYEBCIZMYK7XXVYUQI6ZPTPJ4GZYQ4CKZQ6QYXGTXA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:24.205514540 (59h53m55.395813106s) [noise=true , address=104.36.82.54:4181 , piece_id=3KFMGKO4HP46EEOM4DNTJCEGZEM5JBVCHIFVSUMXVD5R4CPLESVQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:32.926420598 (59h54m4.116719164s) [noise=true , address=104.36.82.56:4246 , piece_id=3JJB4TJZ5ATOK5JX2LDPM3WQ53JTNHIFAV255JHVASY7SQWNEUOQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:36.534683589 (59h54m7.724982162s) [noise=true , address=77.90.31.13:4651 , piece_id=WRVMZBD6MWRLJ7BLGL5W6WPXTADZAEFTJQR3H5WXRAVXMAGDZDMQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:29:57.402874678 (59h54m28.593173267s) [noise=true , address=104.36.82.46:4485 , piece_id=W6SWFXRHMFPM5ZXKJ2FRCUEJEBVLLPLWP54JX5MGA3FGGOHXMLIQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: <nil> Jul 26 23:30:01.517749535 (59h54m32.708048119s) [noise=true , address=104.36.82.47:4551 , piece_id=MO4J3IR3KXGI6BPZ6TLHMOX3ZLWNDL3NIK7NGSZM652DGVDOENVQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518130559 (59h54m32.708429152s) [noise=true , address=77.90.31.15:4726 , piece_id=DCCHPOA4PCJKBSJOKPQIOZ3RBOKLQNFFOM7JJ6DKO5LCWAQUJR4Q, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518398932 (59h54m32.708697513s) [noise=true , address=104.36.82.52:4096 , piece_id=4WMWSOMD5QI3CZJ42PXLQ7MHFZUHWQMLRWWOG2QHIJG4EVUGEUWA, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:01.518639062 (59h54m32.708937643s) [noise=true , address=104.36.82.16:4303 , piece_id=JV4A3YRIO5VJOWSUZECPRS76EFCRR5BKJXKIGPZFCILZTEC2SVSQ, seg_pos={0 0}, upload=57432]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection; write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58736->104.36.82.16:4303: use of closed network connection Jul 26 23:30:01.518706085 (59h54m32.709004678s) [seg_pos={0 0}, upload=57432]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:30:01.661162039 (59h54m32.85146061s) [upload=57432]: Done uploading segments... Jul 26 23:30:01.661222438 (59h54m32.851521019s) [upload=57432]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.146359408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [...X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.114095074s FAIL (2025-07-26 23:29:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.114117844s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.............................X.XX...XX..X..XXXXX....XX.XX..X] duration: 30.000364197s FAIL (2025-07-26 23:29:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000397937s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XX......X.X.......X.......X........XX....X....X.X..X........] duration: 54.767733979s FAIL (2025-07-26 23:29:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.767764012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.......................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.106019644s FAIL (2025-07-26 23:29:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== se=true , address=104.36.82.2:4729 , piece_id=CMX3K76C7SHDIJX2OBFL2U66CM7E67F4JZ3QM2OIRI2K4KNQWTNQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.981900612 (59h53m9.172199191s) [noise=true , address=77.90.31.12:4590 , piece_id=2YQSPGWLHDQQSE3M6PXEVZ5QMMYXSV2B5KEZTA5NPKICFDJB2QSA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.982306567 (59h53m9.172605149s) [noise=true , address=104.36.82.6:4876 , piece_id=XJHPPCDFK3JOIDFF7OTHZKA665FOYBUNZ2UMFQA7O5H5CACAFEWA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.982668788 (59h53m9.172967371s) [noise=true , address=104.36.82.6:4874 , piece_id=OXLY6F3CBX6TH7IAYHHCNRIPKLYXDYRIC377VNTUPOSD2KKWOJIQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.983069062 (59h53m9.173367658s) [noise=true , address=104.36.82.11:4115 , piece_id=OTLRUOGH2YV3EQB746UWAUR47MSGVFXX5H44OYVZOSZATANHS4OA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.983413478 (59h53m9.173712058s) [noise=true , address=45.91.116.8:4047 , piece_id=CM7CPCMFGDHDHX2QL2VDUPZBAMRQL6DOPBAE3X5BILLYHHAPBCWQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.983952470 (59h53m9.17425105s) [noise=true , address=45.91.116.4:4937 , piece_id=AQTCPUJM4FQKPUG5ID2KVRK5ERXOI7R7B2AH45AUXRUF7VHVU6KQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.984064050 (59h53m9.174362629s) [noise=true , address=104.36.82.14:4223 , piece_id=5AFXH7BDF4VUW4UHVUITRYOQUCAWQXBHJMFHIQQUHBNGBO6KJQ5Q, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.984512935 (59h53m9.174811519s) [noise=true , address=45.91.116.12:4225 , piece_id=KQ3T3WVHNDWIMMFYRFPCVMNJ3IXFAHJFYHEIKCY2M26J3JV2U4NA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.984624015 (59h53m9.174922594s) [noise=true , address=104.36.82.2:4732 , piece_id=4R3SR6X3FYOXHWRIQB6RRFYRDZX5EPNFARHBBXZSQMVJRIVRQOFA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.985034602 (59h53m9.175333187s) [noise=true , address=45.91.116.7:4011 , piece_id=626AKXOINBYXB45WJF2PZGZG6YXZY4LPSSBCPKGE35LNNCMLVRZQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.985302917 (59h53m9.175601499s) [noise=true , address=45.91.116.5:4298 , piece_id=VGQDMD43B46D3UM3GEAAMRS76WNKOUXZNHMKYKXHVLAWMEKJFNIA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.986063794 (59h53m9.176362379s) [noise=true , address=104.36.82.4:4804 , piece_id=5TIWDMUOC2UMGBVHTV73CPWYUKXNTQTTYJ6VXKNXHGJK2OSH4YZQ, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:37.986316295 (59h53m9.176614874s) [noise=true , address=104.36.82.4:4801 , piece_id=P2ZLS4LC55237R7GPY7TQTQO56V6J74EMBIUYFJEQFUVOPAAZRCA, seg_pos={0 0}, upload=57423]: Uploading piece... Jul 26 23:28:38.600319812 (59h53m9.790618393s) [upload=57423]: Next returned nil segment Jul 26 23:28:38.600340730 (59h53m9.79063932s) [upload=57423]: Waiting for error group managing segments... Jul 26 23:28:38.739151226 (59h53m9.929449799s) [noise=true , address=45.91.116.12:4225 , piece_id=KQ3T3WVHNDWIMMFYRFPCVMNJ3IXFAHJFYHEIKCY2M26J3JV2U4NA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:38.743130713 (59h53m9.933429291s) [noise=true , address=45.91.116.4:4937 , piece_id=AQTCPUJM4FQKPUG5ID2KVRK5ERXOI7R7B2AH45AUXRUF7VHVU6KQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:38.743242034 (59h53m9.933540606s) [noise=true , address=45.91.116.6:4973 , piece_id=MU7TVKUZHMGLGL5DDPFQN2PTNQZYQN5UJ5LZZ4KRSFKWNCLVINUA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:38.744154338 (59h53m9.934452907s) [noise=true , address=45.91.116.8:4047 , piece_id=CM7CPCMFGDHDHX2QL2VDUPZBAMRQL6DOPBAE3X5BILLYHHAPBCWQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:38.830934642 (59h53m10.021233221s) [noise=true , address=45.91.116.7:4011 , piece_id=626AKXOINBYXB45WJF2PZGZG6YXZY4LPSSBCPKGE35LNNCMLVRZQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:39.181222061 (59h53m10.37152064s) [noise=true , address=45.91.116.5:4298 , piece_id=VGQDMD43B46D3UM3GEAAMRS76WNKOUXZNHMKYKXHVLAWMEKJFNIA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:28:57.557411544 (59h53m28.747710136s) [noise=true , address=104.36.82.16:4294 , piece_id=LW6QYPWGPEMUUEYQPJLS5AV2SF3I37J5424HVHGPFROBHUSY5LEA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: <nil> Jul 26 23:29:07.596796186 (59h53m38.787094755s) [noise=true , address=104.36.82.2:4729 , piece_id=CMX3K76C7SHDIJX2OBFL2U66CM7E67F4JZ3QM2OIRI2K4KNQWTNQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:29:07.596950212 (59h53m38.787248793s) [noise=true , address=104.36.82.11:4115 , piece_id=OTLRUOGH2YV3EQB746UWAUR47MSGVFXX5H44OYVZOSZATANHS4OA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 139.59.144.22:52888->104.36.82.11:4115: use of closed network connection; write tcp 139.59.144.22:52888->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52888->104.36.82.11:4115: use of closed network connection Jul 26 23:29:07.597033433 (59h53m38.787332013s) [noise=true , address=104.36.82.4:4801 , piece_id=P2ZLS4LC55237R7GPY7TQTQO56V6J74EMBIUYFJEQFUVOPAAZRCA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:35594->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:35594->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35594->104.36.82.4:4801: use of closed network connection Jul 26 23:29:07.597111334 (59h53m38.787409916s) [noise=true , address=104.36.82.6:4874 , piece_id=OXLY6F3CBX6TH7IAYHHCNRIPKLYXDYRIC377VNTUPOSD2KKWOJIQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:40120->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:40120->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40120->104.36.82.6:4874: use of closed network connection Jul 26 23:29:07.597176689 (59h53m38.787475262s) [noise=true , address=104.36.82.6:4876 , piece_id=XJHPPCDFK3JOIDFF7OTHZKA665FOYBUNZ2UMFQA7O5H5CACAFEWA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:36540->104.36.82.6:4876: use of closed network connection; write tcp 139.59.144.22:36540->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36540->104.36.82.6:4876: use of closed network connection Jul 26 23:29:07.597186733 (59h53m38.787485303s) [noise=true , address=104.36.82.14:4223 , piece_id=5AFXH7BDF4VUW4UHVUITRYOQUCAWQXBHJMFHIQQUHBNGBO6KJQ5Q, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:41112->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:41112->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41112->104.36.82.14:4223: use of closed network connection Jul 26 23:29:07.597247502 (59h53m38.787546071s) [noise=true , address=104.36.82.2:4732 , piece_id=4R3SR6X3FYOXHWRIQB6RRFYRDZX5EPNFARHBBXZSQMVJRIVRQOFA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 139.59.144.22:44386->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:44386->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 26 23:29:07.597253915 (59h53m38.787552488s) [noise=true , address=104.36.82.4:4804 , piece_id=5TIWDMUOC2UMGBVHTV73CPWYUKXNTQTTYJ6VXKNXHGJK2OSH4YZQ, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 139.59.144.22:53036->104.36.82.4:4804: use of closed network connection; write tcp 139.59.144.22:53036->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53036->104.36.82.4:4804: use of closed network connection Jul 26 23:29:07.597303947 (59h53m38.787602516s) [noise=true , address=77.90.31.12:4590 , piece_id=2YQSPGWLHDQQSE3M6PXEVZ5QMMYXSV2B5KEZTA5NPKICFDJB2QSA, seg_pos={0 0}, upload=57423]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:54370->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:07.597350903 (59h53m38.787649481s) [seg_pos={0 0}, upload=57423]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:29:07.700758084 (59h53m38.891056656s) [upload=57423]: Done uploading segments... Jul 26 23:29:07.700785347 (59h53m38.891083926s) [upload=57423]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.106044556s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............................X.XX...XX..X..XXXXX....XX.XX..] duration: 9.869341402s FAIL (2025-07-26 23:28:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.869363077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....X] duration: 1m0.106509581s FAIL (2025-07-26 23:28:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== em_ObjectBegin Jul 26 23:27:47.292057116 (59h52m18.482355699s) [upload=57414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:27:47.502728833 (59h52m18.693027415s) [seg_pos={0 0}, upload=57414]: Begin upload segment... Jul 26 23:27:47.502803053 (59h52m18.693101622s) [seg_pos={0 0}, upload=57414]: Done begin upload segment. Jul 26 23:27:47.502859147 (59h52m18.693157738s) [noise=true , address=45.91.116.13:4175 , piece_id=6GPQ3KXBYBN55MDLQIB2JGFYYSRZ26AEEJIVBE5CMLTUMQV5WDJA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.503339875 (59h52m18.693638458s) [noise=true , address=77.90.31.12:4602 , piece_id=B4OJARXRTVEIJWDAT4Q7P6A2QTROUECIBHW5GMXXK5GGPGLRXAEQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.503715583 (59h52m18.694014168s) [noise=true , address=79.127.201.215:4349 , piece_id=7JW3RYABZWYTXNYF3FRNAACNPH2NZBOX5C3C46IUCYE5WEQBEA5Q, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.504113132 (59h52m18.694411718s) [noise=true , address=104.36.82.47:4549 , piece_id=F7T52TNCAQNLJ2BRJF7OQXZY3PBJUXTOJBZXWJ4PNV35US4E2IUA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.504711094 (59h52m18.695009675s) [noise=true , address=64.49.13.140:4508 , piece_id=OF5OGJASQSYPHTKR3TIRWW5KSTZ5OK7RLFUWIRIGRHT4QJ4Z2OFQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.505088743 (59h52m18.695387328s) [noise=true , address=79.127.205.234:4604 , piece_id=O4P7TKQQJ3DDNN5TDWC577JV7N4DH2ZR5E3NSOM6FHUTUCZIM5AA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.505414072 (59h52m18.695712657s) [noise=true , address=77.90.31.21:4887 , piece_id=YHANZ32ROOKEA6TDF733RJCMBZH6SLH4TAHOHSXUEIJQF64EIVXQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.505741810 (59h52m18.696040396s) [noise=true , address=77.90.31.13:4659 , piece_id=JCPVFOJLZEIORKJ7APUL2U3ZYTO5RL5ULQ4IXSIIOP4ZOZYYVWNQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.506022376 (59h52m18.696321003s) [noise=true , address=79.127.220.101:4041 , piece_id=IZNBZ47TFXBDW7IE5ZIMWEIEQPCJTDI4D2I724PNNSM6L5FVKLEA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.506233786 (59h52m18.696532365s) [noise=true , address=45.91.116.11:4121 , piece_id=HKHIQHOPCXCPTBMRSWYLTD3PGF7IJ75ELWCCREOKTXJ4XAK32NAQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.506522952 (59h52m18.696821538s) [noise=true , address=104.36.82.53:4150 , piece_id=VLQZRHQOMDWGOSOHGTY6F22NLILVWMQUV66HOEOASFGVQ3NP63TQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.506959732 (59h52m18.697258322s) [noise=true , address=79.127.205.246:4282 , piece_id=757YFITE6BMCVI2CWRRK6D4TP5N3VJ7OVQCKAD6QTOZWRBQ2WQ2A, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.507419635 (59h52m18.697718224s) [noise=true , address=79.127.205.235:4678 , piece_id=HUP2RRPBQUYVPEUDLIJD5STI5TGKTZRGZOCVCRTRG6CY2HPPFQCA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.507894368 (59h52m18.698192954s) [noise=true , address=79.127.219.51:4210 , piece_id=HBTZJVXLPUHE3322Y6YVBKJWLTWBSJSNKKRU2PYQF2HPS4ILTUQQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.508256425 (59h52m18.698555015s) [noise=true , address=77.90.31.11:4560 , piece_id=TDWW3FIKJ524WPD4NXS4LF5WWEHTENWA6STFZJ3EFO3XKV7BZBSA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.508756437 (59h52m18.699055025s) [noise=true , address=104.36.82.52:4089 , piece_id=HJM7VMNX5IL2YFQJR4MU2E5VFPE4YBQOZWIYOZS5S2GQHVZSOADQ, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.509077788 (59h52m18.699376374s) [noise=true , address=45.91.116.5:4306 , piece_id=2K5PG6ZAPE5ATI3DF7ACZ63QAFMF4WJZ2GU2Y3DMVRQXYGRCPP4A, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.509355641 (59h52m18.699654224s) [noise=true , address=79.127.220.98:4422 , piece_id=UR56S3X7QCSWTEFHUWEBD4INHHY335FRVMA6DNYZQJ7H4TEENPWA, seg_pos={0 0}, upload=57414]: Uploading piece... Jul 26 23:27:47.975042051 (59h52m19.16534063s) [noise=true , address=79.127.220.98:4422 , piece_id=UR56S3X7QCSWTEFHUWEBD4INHHY335FRVMA6DNYZQJ7H4TEENPWA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE, address:79.127.220.98:4422): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.020885776 (59h52m19.211184361s) [noise=true , address=79.127.220.101:4041 , piece_id=IZNBZ47TFXBDW7IE5ZIMWEIEQPCJTDI4D2I724PNNSM6L5FVKLEA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.050091580 (59h52m19.240390159s) [noise=true , address=79.127.219.51:4210 , piece_id=HBTZJVXLPUHE3322Y6YVBKJWLTWBSJSNKKRU2PYQF2HPS4ILTUQQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.082126631 (59h52m19.272425213s) [noise=true , address=45.91.116.13:4175 , piece_id=6GPQ3KXBYBN55MDLQIB2JGFYYSRZ26AEEJIVBE5CMLTUMQV5WDJA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.087048757 (59h52m19.277347323s) [upload=57414]: Next returned nil segment Jul 26 23:27:48.087067047 (59h52m19.277365613s) [upload=57414]: Waiting for error group managing segments... Jul 26 23:27:48.233982057 (59h52m19.424280629s) [noise=true , address=77.90.31.21:4887 , piece_id=YHANZ32ROOKEA6TDF733RJCMBZH6SLH4TAHOHSXUEIJQF64EIVXQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:27:48.397043554 (59h52m19.587342124s) [noise=true , address=64.49.13.140:4508 , piece_id=OF5OGJASQSYPHTKR3TIRWW5KSTZ5OK7RLFUWIRIGRHT4QJ4Z2OFQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.435334605 (59h52m19.625633188s) [noise=true , address=45.91.116.11:4121 , piece_id=HKHIQHOPCXCPTBMRSWYLTD3PGF7IJ75ELWCCREOKTXJ4XAK32NAQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:48.771758835 (59h52m19.962057408s) [noise=true , address=79.127.205.235:4678 , piece_id=HUP2RRPBQUYVPEUDLIJD5STI5TGKTZRGZOCVCRTRG6CY2HPPFQCA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:27:48.820492121 (59h52m20.010790698s) [noise=true , address=79.127.205.234:4604 , piece_id=O4P7TKQQJ3DDNN5TDWC577JV7N4DH2ZR5E3NSOM6FHUTUCZIM5AA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:27:48.831436149 (59h52m20.021734719s) [noise=true , address=79.127.205.246:4282 , piece_id=757YFITE6BMCVI2CWRRK6D4TP5N3VJ7OVQCKAD6QTOZWRBQ2WQ2A, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:27:52.386320602 (59h52m23.576619196s) [noise=true , address=45.91.116.5:4306 , piece_id=2K5PG6ZAPE5ATI3DF7ACZ63QAFMF4WJZ2GU2Y3DMVRQXYGRCPP4A, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: storage node overloaded, request limit: 2000 Jul 26 23:28:04.806707179 (59h52m35.997005763s) [noise=true , address=79.127.201.215:4349 , piece_id=7JW3RYABZWYTXNYF3FRNAACNPH2NZBOX5C3C46IUCYE5WEQBEA5Q, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:28:04.918887114 (59h52m36.109185695s) [noise=true , address=77.90.31.13:4659 , piece_id=JCPVFOJLZEIORKJ7APUL2U3ZYTO5RL5ULQ4IXSIIOP4ZOZYYVWNQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:28:12.048856721 (59h52m43.239155294s) [noise=true , address=104.36.82.53:4150 , piece_id=VLQZRHQOMDWGOSOHGTY6F22NLILVWMQUV66HOEOASFGVQ3NP63TQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:28:18.761142802 (59h52m49.951441373s) [noise=true , address=104.36.82.52:4089 , piece_id=HJM7VMNX5IL2YFQJR4MU2E5VFPE4YBQOZWIYOZS5S2GQHVZSOADQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:28:46.273322211 (59h53m17.463620786s) [noise=true , address=104.36.82.47:4549 , piece_id=F7T52TNCAQNLJ2BRJF7OQXZY3PBJUXTOJBZXWJ4PNV35US4E2IUA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: <nil> Jul 26 23:28:47.023369168 (59h53m18.213667746s) [noise=true , address=77.90.31.12:4602 , piece_id=B4OJARXRTVEIJWDAT4Q7P6A2QTROUECIBHW5GMXXK5GGPGLRXAEQ, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:47.023599949 (59h53m18.213898528s) [noise=true , address=77.90.31.11:4560 , piece_id=TDWW3FIKJ524WPD4NXS4LF5WWEHTENWA6STFZJ3EFO3XKV7BZBSA, seg_pos={0 0}, upload=57414]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:28:47.023632528 (59h53m18.213931109s) [seg_pos={0 0}, upload=57414]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:28:47.127517030 (59h53m18.317815613s) [upload=57414]: Done uploading segments... Jul 26 23:28:47.127546348 (59h53m18.31784493s) [upload=57414]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 1m0.106542194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [....X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.] duration: 28.038502259s FAIL (2025-07-26 23:28:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 28.038539292s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.XX......X.X.......X.......X........XX....X....X.X..X.......] duration: 57.069726154s FAIL (2025-07-26 23:28:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.069745028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............................X.XX...XX..X..XXXXX....XX.XX.] duration: 21.100368856s FAIL (2025-07-26 23:28:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.100394274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [.....X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.105053595s FAIL (2025-07-26 23:28:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.105083758s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [........................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.116832855s FAIL (2025-07-26 23:28:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== : op; piecestore: piecestore close: conn closed: op Jul 26 23:26:48.170247498 (59h51m19.360546087s) [seg_pos={0 0}, upload=57391]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:26:48.273993272 (59h51m19.464291862s) [upload=57391]: Done uploading segments... Jul 26 23:26:48.274041588 (59h51m19.464340179s) [upload=57391]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:27:31.203957644 (59h52m2.394256224s) [upload=57407]: Starting upload Jul 26 23:27:31.204040073 (59h52m2.394338655s) [upload=57407]: Done starting upload Jul 26 23:27:31.204175821 (59h52m2.394474402s) [upload=57407]: Uploading segments... Jul 26 23:27:31.204294092 (59h52m2.394592672s) [upload=57407]: Got next segment. Inline: false Jul 26 23:27:31.204335483 (59h52m2.394634063s) [upload=57407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:27:31.204341694 (59h52m2.394640273s) [upload=57407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:27:31.319565507 (59h52m2.509864093s) [seg_pos={0 0}, upload=57407]: Begin upload segment... Jul 26 23:27:31.319651590 (59h52m2.509950179s) [noise=true , address=104.36.82.15:4257 , piece_id=X54IAKIW5N35OGY22NMMFEHSC3TBPWPK4THIIFBS323IDUUCY4FQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.320304668 (59h52m2.510603257s) [noise=true , address=45.91.116.7:4011 , piece_id=ZOWSQ47SSC7ODCMBQITMF6OXZINMDOQZSQOAHVOKSTNBC3NVMFFQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.320997563 (59h52m2.511296153s) [noise=true , address=45.91.116.9:4262 , piece_id=DZ5KVMHP4CZ7TSILRJFE2W6GXPOMMIKAE6QTHQ6YYQV2AFEUNNAQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.321539271 (59h52m2.511837863s) [seg_pos={0 0}, upload=57407]: Done begin upload segment. Jul 26 23:27:31.321574387 (59h52m2.511872978s) [noise=true , address=77.90.31.18:4807 , piece_id=2WXUXAGKVZI2HOD65CNGAZYDSF2RQHGBITCCUYMQKBHLIP2QQYBQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.326298617 (59h52m2.516597189s) [noise=true , address=77.90.31.12:4590 , piece_id=TR2JZ5WLAMQTXDBQHI4FTUKOWA7FELNVTQFX3RRVKFFYOETOH7GA, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.326735402 (59h52m2.517033983s) [noise=true , address=104.36.82.16:4295 , piece_id=SAZN7TZWZZNROAZNWKJ36AMSHSZSMCSIA2SNOBSOYPS34HFVV43Q, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.327030909 (59h52m2.517329499s) [noise=true , address=104.36.82.9:4006 , piece_id=6ZGJ6G5G4URTG3SWFQGHF34TRWBORZLCD46HAZ43BT5E4SOXPZHQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.327374437 (59h52m2.517673029s) [noise=true , address=104.36.82.5:4837 , piece_id=5ZGIPAD4FSCMLO5N3CLCJNDRDS2FHOTKPU6XZ45DH74PG7MIIA5A, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.327770241 (59h52m2.518068829s) [noise=true , address=104.36.82.6:4874 , piece_id=7O6XIIWZZ6FZVIU7ARGNTYHJN7E3TDDNL2SUKW5FQPD65FGGKAXA, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.328091278 (59h52m2.518389857s) [noise=true , address=104.36.82.10:4077 , piece_id=E62SY5NVKZN62A3B4HUNSNZLJN5KP4Z2IESQKUY3OHN2NZUSFF4Q, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.328527353 (59h52m2.518825934s) [noise=true , address=45.91.116.10:4081 , piece_id=UVSFV3TRZ4SFCFBI7RLLODEIL5YEUPBBP4PCBGZWJSYJIQSTCG6A, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.328839136 (59h52m2.519137718s) [noise=true , address=45.91.116.13:4155 , piece_id=7MVFHPM7QQG3HB4FE7PBGJAJAPTL4ED5YJS5WEW7B4QMCQMAUDYQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.329145715 (59h52m2.519444299s) [noise=true , address=45.91.116.5:4297 , piece_id=736B76WHL4NTMJPBGTPKHYDIAZFTSTSMQ47NACDONLKFWLSHWNIA, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.329431498 (59h52m2.519730077s) [noise=true , address=45.91.116.8:4048 , piece_id=7SL32HUGJ6DNBEGSTR2XZSY7LN5WFDNZPPKLRRUDN3RXTIE2ABRA, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.329748999 (59h52m2.520047587s) [noise=true , address=104.36.82.8:4971 , piece_id=PDY54YR3YYXSPJO57XLVDNGCX5NUTCPITEPMCA7RSTP7QZOHXDHQ, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.330305718 (59h52m2.520604304s) [noise=true , address=77.90.31.14:4661 , piece_id=ISJBJZV5K2XR4J26TJDS3HQNIMTHMNSQVS243RYYAT3TXXSNW3EA, seg_pos={0 0}, upload=57407]: Uploading piece... Jul 26 23:27:31.931872887 (59h52m3.122171468s) [upload=57407]: Next returned nil segment Jul 26 23:27:31.931894292 (59h52m3.122192871s) [upload=57407]: Waiting for error group managing segments... Jul 26 23:27:32.070400709 (59h52m3.26069929s) [noise=true , address=45.91.116.10:4081 , piece_id=UVSFV3TRZ4SFCFBI7RLLODEIL5YEUPBBP4PCBGZWJSYJIQSTCG6A, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.077692624 (59h52m3.267991204s) [noise=true , address=45.91.116.8:4048 , piece_id=7SL32HUGJ6DNBEGSTR2XZSY7LN5WFDNZPPKLRRUDN3RXTIE2ABRA, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.080862208 (59h52m3.271160778s) [noise=true , address=45.91.116.13:4155 , piece_id=7MVFHPM7QQG3HB4FE7PBGJAJAPTL4ED5YJS5WEW7B4QMCQMAUDYQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.094652098 (59h52m3.284950689s) [noise=true , address=45.91.116.7:4011 , piece_id=ZOWSQ47SSC7ODCMBQITMF6OXZINMDOQZSQOAHVOKSTNBC3NVMFFQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.096901305 (59h52m3.287199874s) [noise=true , address=77.90.31.14:4661 , piece_id=ISJBJZV5K2XR4J26TJDS3HQNIMTHMNSQVS243RYYAT3TXXSNW3EA, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.237449238 (59h52m3.427747807s) [noise=true , address=45.91.116.5:4297 , piece_id=736B76WHL4NTMJPBGTPKHYDIAZFTSTSMQ47NACDONLKFWLSHWNIA, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:32.368782345 (59h52m3.55908091s) [noise=true , address=45.91.116.9:4262 , piece_id=DZ5KVMHP4CZ7TSILRJFE2W6GXPOMMIKAE6QTHQ6YYQV2AFEUNNAQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:33.525462043 (59h52m4.715760629s) [noise=true , address=77.90.31.12:4590 , piece_id=TR2JZ5WLAMQTXDBQHI4FTUKOWA7FELNVTQFX3RRVKFFYOETOH7GA, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:41.840051544 (59h52m13.03035012s) [noise=true , address=77.90.31.18:4807 , piece_id=2WXUXAGKVZI2HOD65CNGAZYDSF2RQHGBITCCUYMQKBHLIP2QQYBQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:27:58.649691795 (59h52m29.839990375s) [noise=true , address=104.36.82.15:4257 , piece_id=X54IAKIW5N35OGY22NMMFEHSC3TBPWPK4THIIFBS323IDUUCY4FQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: <nil> Jul 26 23:28:00.906390868 (59h52m32.096689447s) [noise=true , address=104.36.82.5:4837 , piece_id=5ZGIPAD4FSCMLO5N3CLCJNDRDS2FHOTKPU6XZ45DH74PG7MIIA5A, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:00.906549292 (59h52m32.096847883s) [noise=true , address=104.36.82.8:4971 , piece_id=PDY54YR3YYXSPJO57XLVDNGCX5NUTCPITEPMCA7RSTP7QZOHXDHQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:00.906626008 (59h52m32.096924586s) [noise=true , address=104.36.82.10:4077 , piece_id=E62SY5NVKZN62A3B4HUNSNZLJN5KP4Z2IESQKUY3OHN2NZUSFF4Q, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:00.906723646 (59h52m32.097022225s) [noise=true , address=104.36.82.16:4295 , piece_id=SAZN7TZWZZNROAZNWKJ36AMSHSZSMCSIA2SNOBSOYPS34HFVV43Q, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:00.906846110 (59h52m32.09714469s) [noise=true , address=104.36.82.9:4006 , piece_id=6ZGJ6G5G4URTG3SWFQGHF34TRWBORZLCD46HAZ43BT5E4SOXPZHQ, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:42408->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:42408->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42408->104.36.82.9:4006: use of closed network connection Jul 26 23:28:00.906921276 (59h52m32.097219853s) [noise=true , address=104.36.82.6:4874 , piece_id=7O6XIIWZZ6FZVIU7ARGNTYHJN7E3TDDNL2SUKW5FQPD65FGGKAXA, seg_pos={0 0}, upload=57407]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:49016->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:49016->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49016->104.36.82.6:4874: use of closed network connection Jul 26 23:28:00.906938015 (59h52m32.097236608s) [seg_pos={0 0}, upload=57407]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:28:01.020769616 (59h52m32.211068189s) [upload=57407]: Done uploading segments... Jul 26 23:28:01.020822991 (59h52m32.211121559s) [upload=57407]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.116860343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....] duration: 47.111997311s FAIL (2025-07-26 23:27:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:15.489277987 (59h46m46.679576569s) [upload=57325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:15.702839669 (59h46m46.893138255s) [seg_pos={0 0}, upload=57325]: Begin upload segment... Jul 26 23:22:15.702913804 (59h46m46.893212391s) [seg_pos={0 0}, upload=57325]: Done begin upload segment. Jul 26 23:22:15.702962240 (59h46m46.893260825s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703624299 (59h46m46.893922886s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703705288 (59h46m46.89400388s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704717183 (59h46m46.895015774s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704911246 (59h46m46.895209836s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705058351 (59h46m46.895356933s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705963686 (59h46m46.896262277s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706013141 (59h46m46.896311734s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706883998 (59h46m46.897182585s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707155160 (59h46m46.89745375s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707530555 (59h46m46.897829145s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708053028 (59h46m46.898351621s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708475199 (59h46m46.89877379s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709329053 (59h46m46.899627645s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709728548 (59h46m46.900027137s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710158627 (59h46m46.900457217s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710555627 (59h46m46.900854216s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710989594 (59h46m46.901288183s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:16.267925926 (59h46m47.458224496s) [upload=57325]: Next returned nil segment Jul 26 23:22:16.267944613 (59h46m47.458243195s) [upload=57325]: Waiting for error group managing segments... Jul 26 23:22:16.480904398 (59h46m47.671202983s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.532367572 (59h46m47.72266619s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.558052644 (59h46m47.748351223s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.945385561 (59h46m48.13568414s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.179396266 (59h46m48.36969484s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.301285788 (59h46m48.491584368s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.602033920 (59h46m48.7923325s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:20.039346202 (59h46m51.229644789s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:32.674372421 (59h47m3.864670986s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:34.394451336 (59h47m5.584749909s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:39.514179337 (59h47m10.704477914s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:51.857893010 (59h47m23.048191575s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:23:15.210646325 (59h47m46.400944904s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210780376 (59h47m46.401078957s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210936257 (59h47m46.401234841s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: expected piece hash; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; EOF Jul 26 23:23:15.211100863 (59h47m46.401399445s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.211326802 (59h47m46.401625387s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection Jul 26 23:23:15.211574205 (59h47m46.401872789s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:15.211611872 (59h47m46.401910454s) [seg_pos={0 0}, upload=57325]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:23:15.320657081 (59h47m46.510955693s) [upload=57325]: Done uploading segments... Jul 26 23:23:15.320691301 (59h47m46.510989891s) [upload=57325]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 47.112032183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.XX......X.X.......X.......X........XX....X....X.X..X......] duration: 55.756121817s FAIL (2025-07-26 23:27:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.756152182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................................X.XX...XX..X..XXXXX....XX.XX] duration: 30.000674375s FAIL (2025-07-26 23:27:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000702751s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [......X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 30.11199534s FAIL (2025-07-26 23:27:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.112023184s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.........................X......XXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.106075419s FAIL (2025-07-26 23:26:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3 but got 8) Jul 26 23:26:18.446850516 (59h50m49.637149101s) [upload=57391]: Starting upload Jul 26 23:26:18.446913717 (59h50m49.637212283s) [upload=57391]: Done starting upload Jul 26 23:26:18.447052306 (59h50m49.637350887s) [upload=57391]: Uploading segments... Jul 26 23:26:18.447186237 (59h50m49.637484826s) [upload=57391]: Got next segment. Inline: false Jul 26 23:26:18.447286088 (59h50m49.637584659s) [upload=57391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:26:18.447293215 (59h50m49.637591794s) [upload=57391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:26:18.696929402 (59h50m49.887227999s) [seg_pos={0 0}, upload=57391]: Begin upload segment... Jul 26 23:26:18.696983494 (59h50m49.887282059s) [seg_pos={0 0}, upload=57391]: Done begin upload segment. Jul 26 23:26:18.697015314 (59h50m49.887313899s) [noise=true , address=45.91.116.4:4937 , piece_id=MTIM6343C7CNZS5VHVNMLCSH3TVLSZFQ4ZDOIDYE2CBL6XWHYSVA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.697030918 (59h50m49.887329497s) [noise=true , address=104.36.82.3:4767 , piece_id=DBCOEF2DWU7IFXNJ6TJNBTWLT2DYGW3Z4XEPRLLOOIIF422E7REQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.697388696 (59h50m49.887687281s) [noise=true , address=45.91.116.12:4227 , piece_id=6NJRPFZZOTLFJW3NWMS7ZC5XDX7IFFCP6R25I4KSHRBH5ZQXVB5A, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.697438692 (59h50m49.887737274s) [noise=true , address=104.36.82.9:4007 , piece_id=3JPAHASNQH6HU4MOLAEHAYBXXQ5JFMG5LZFQXXE4S4O3C3T3BPXQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.697695160 (59h50m49.887993741s) [noise=true , address=45.91.116.10:4082 , piece_id=N47EFVUOTKXVVDZSCX4RHZ3REYSUWMNZUCYLBBABPGXPJHHUS2XA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.697727809 (59h50m49.888026389s) [noise=true , address=45.91.116.13:4155 , piece_id=3NT6NHSCUH27PJXHOR5YLQN5224KGHB5PVJXZJ74X4QMAYALKECQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.698044380 (59h50m49.888342962s) [noise=true , address=77.90.31.19:4843 , piece_id=ZFA4SAN2EBU54D2TK4RQHVDWI4GJJQJGJX5ZPI4XTOHNLHA7K3JQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.698435780 (59h50m49.888734365s) [noise=true , address=45.91.116.14:4190 , piece_id=CFBIGZE2IFMI7VFPUEAD5DNYLQC6YHCKZGHDJU3DGBIJNGSFFCDA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.698761816 (59h50m49.889060398s) [noise=true , address=104.36.82.16:4295 , piece_id=CAZ2IB6Y4Y547VV6T7K4ZFWO33SXGKIKYNU6XKMRAEKN7EYHEAKA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.698848298 (59h50m49.889146879s) [noise=true , address=45.91.116.7:4010 , piece_id=K47SP7GI5RCCBJPDLGLHFYTUH34B7QHXW3F5LUR2TR7MYTVCJITA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.699076987 (59h50m49.889375569s) [noise=true , address=77.90.31.14:4661 , piece_id=LWIHUG6WDJUA2FYTIBVISN3QT54FE6VGE475IYCXKBIPSXUT7FOA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.699385379 (59h50m49.889683947s) [noise=true , address=104.36.82.13:4186 , piece_id=BQVNAT53HYWDOD4VNJNQUPPBWG5YWU4NHYKCG5AFUWGSUGLFOUSQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.699568369 (59h50m49.889866951s) [noise=true , address=104.36.82.12:4150 , piece_id=5UP7WGDGDQHG6MNO3LP3LGDYWQ7GDVKG5HYC7F4JUYZWYERMSZMA, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.699854125 (59h50m49.89015271s) [noise=true , address=104.36.82.9:4005 , piece_id=N36I5IIB4T4HQXFRP7D3SBYSTDUZ2EF4PCTZM4NPRKDLKEVPDWKQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.700093280 (59h50m49.890391864s) [noise=true , address=104.36.82.5:4838 , piece_id=MPFJVT5NH35FALUL3TP4IGT6O7RWP3HT34BN4SF4PNBNRVSPOM4A, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:18.700515485 (59h50m49.890814072s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKYVPJEYNJHWZV3LDXN6FBI6ZZWX3U5O44FQOFTJMWGI7J4YMLQ, seg_pos={0 0}, upload=57391]: Uploading piece... Jul 26 23:26:19.309938106 (59h50m50.500236671s) [upload=57391]: Next returned nil segment Jul 26 23:26:19.309957653 (59h50m50.500256238s) [upload=57391]: Waiting for error group managing segments... Jul 26 23:26:19.439393827 (59h50m50.629692404s) [noise=true , address=45.91.116.13:4155 , piece_id=3NT6NHSCUH27PJXHOR5YLQN5224KGHB5PVJXZJ74X4QMAYALKECQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.440047079 (59h50m50.630345657s) [noise=true , address=45.91.116.7:4010 , piece_id=K47SP7GI5RCCBJPDLGLHFYTUH34B7QHXW3F5LUR2TR7MYTVCJITA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.446188425 (59h50m50.636487005s) [noise=true , address=45.91.116.10:4082 , piece_id=N47EFVUOTKXVVDZSCX4RHZ3REYSUWMNZUCYLBBABPGXPJHHUS2XA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.454973742 (59h50m50.645272329s) [noise=true , address=45.91.116.14:4190 , piece_id=CFBIGZE2IFMI7VFPUEAD5DNYLQC6YHCKZGHDJU3DGBIJNGSFFCDA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.459221571 (59h50m50.649520141s) [noise=true , address=77.90.31.14:4661 , piece_id=LWIHUG6WDJUA2FYTIBVISN3QT54FE6VGE475IYCXKBIPSXUT7FOA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.462272565 (59h50m50.652571137s) [noise=true , address=45.91.116.12:4227 , piece_id=6NJRPFZZOTLFJW3NWMS7ZC5XDX7IFFCP6R25I4KSHRBH5ZQXVB5A, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.463506496 (59h50m50.65380508s) [noise=true , address=45.91.116.4:4937 , piece_id=MTIM6343C7CNZS5VHVNMLCSH3TVLSZFQ4ZDOIDYE2CBL6XWHYSVA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:19.471522968 (59h50m50.661821541s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKYVPJEYNJHWZV3LDXN6FBI6ZZWX3U5O44FQOFTJMWGI7J4YMLQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:20.828084989 (59h50m52.018383582s) [noise=true , address=77.90.31.19:4843 , piece_id=ZFA4SAN2EBU54D2TK4RQHVDWI4GJJQJGJX5ZPI4XTOHNLHA7K3JQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:39.599201275 (59h51m10.789499857s) [noise=true , address=104.36.82.9:4005 , piece_id=N36I5IIB4T4HQXFRP7D3SBYSTDUZ2EF4PCTZM4NPRKDLKEVPDWKQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:46.312583332 (59h51m17.502881923s) [noise=true , address=104.36.82.9:4007 , piece_id=3JPAHASNQH6HU4MOLAEHAYBXXQ5JFMG5LZFQXXE4S4O3C3T3BPXQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: <nil> Jul 26 23:26:48.169820922 (59h51m19.360119515s) [noise=true , address=104.36.82.13:4186 , piece_id=BQVNAT53HYWDOD4VNJNQUPPBWG5YWU4NHYKCG5AFUWGSUGLFOUSQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:54350->104.36.82.13:4186: use of closed network connection; write tcp 139.59.144.22:54350->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54350->104.36.82.13:4186: use of closed network connection Jul 26 23:26:48.170039008 (59h51m19.360337578s) [noise=true , address=104.36.82.16:4295 , piece_id=CAZ2IB6Y4Y547VV6T7K4ZFWO33SXGKIKYNU6XKMRAEKN7EYHEAKA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 139.59.144.22:39644->104.36.82.16:4295: use of closed network connection; write tcp 139.59.144.22:39644->104.36.82.16:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39644->104.36.82.16:4295: use of closed network connection Jul 26 23:26:48.170106860 (59h51m19.360405427s) [noise=true , address=104.36.82.12:4150 , piece_id=5UP7WGDGDQHG6MNO3LP3LGDYWQ7GDVKG5HYC7F4JUYZWYERMSZMA, seg_pos={0 0}, upload=57391]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:42954->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:42954->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42954->104.36.82.12:4150: use of closed network connection Jul 26 23:26:48.170150937 (59h51m19.360449507s) [noise=true , address=104.36.82.3:4767 , piece_id=DBCOEF2DWU7IFXNJ6TJNBTWLT2DYGW3Z4XEPRLLOOIIF422E7REQ, seg_pos={0 0}, upload=57391]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:57526->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:57526->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57526->104.36.82.3:4767: use of closed network connection Jul 26 23:26:48.170206798 (59h51m19.360505382s) [noise=true , address=104.36.82.5:4838 , piece_id=MPFJVT5NH35FALUL3TP4IGT6O7RWP3HT34BN4SF4PNBNRVSPOM4A, seg_pos={0 0}, upload=57391]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: read tcp 139.59.144.22:36280->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:48.170247498 (59h51m19.360546087s) [seg_pos={0 0}, upload=57391]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:26:48.273993272 (59h51m19.464291862s) [upload=57391]: Done uploading segments... Jul 26 23:26:48.274041588 (59h51m19.464340179s) [upload=57391]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.106107168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X....] duration: 51.428072662s FAIL (2025-07-26 23:26:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:15.489277987 (59h46m46.679576569s) [upload=57325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:15.702839669 (59h46m46.893138255s) [seg_pos={0 0}, upload=57325]: Begin upload segment... Jul 26 23:22:15.702913804 (59h46m46.893212391s) [seg_pos={0 0}, upload=57325]: Done begin upload segment. Jul 26 23:22:15.702962240 (59h46m46.893260825s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703624299 (59h46m46.893922886s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703705288 (59h46m46.89400388s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704717183 (59h46m46.895015774s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704911246 (59h46m46.895209836s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705058351 (59h46m46.895356933s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705963686 (59h46m46.896262277s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706013141 (59h46m46.896311734s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706883998 (59h46m46.897182585s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707155160 (59h46m46.89745375s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707530555 (59h46m46.897829145s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708053028 (59h46m46.898351621s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708475199 (59h46m46.89877379s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709329053 (59h46m46.899627645s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709728548 (59h46m46.900027137s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710158627 (59h46m46.900457217s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710555627 (59h46m46.900854216s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710989594 (59h46m46.901288183s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:16.267925926 (59h46m47.458224496s) [upload=57325]: Next returned nil segment Jul 26 23:22:16.267944613 (59h46m47.458243195s) [upload=57325]: Waiting for error group managing segments... Jul 26 23:22:16.480904398 (59h46m47.671202983s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.532367572 (59h46m47.72266619s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.558052644 (59h46m47.748351223s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.945385561 (59h46m48.13568414s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.179396266 (59h46m48.36969484s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.301285788 (59h46m48.491584368s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.602033920 (59h46m48.7923325s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:20.039346202 (59h46m51.229644789s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:32.674372421 (59h47m3.864670986s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:34.394451336 (59h47m5.584749909s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:39.514179337 (59h47m10.704477914s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:51.857893010 (59h47m23.048191575s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:23:15.210646325 (59h47m46.400944904s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210780376 (59h47m46.401078957s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210936257 (59h47m46.401234841s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: expected piece hash; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; EOF Jul 26 23:23:15.211100863 (59h47m46.401399445s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.211326802 (59h47m46.401625387s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection Jul 26 23:23:15.211574205 (59h47m46.401872789s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:15.211611872 (59h47m46.401910454s) [seg_pos={0 0}, upload=57325]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:23:15.320657081 (59h47m46.510955693s) [upload=57325]: Done uploading segments... Jul 26 23:23:15.320691301 (59h47m46.510989891s) [upload=57325]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 51.428102862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.................................X.XX...XX..X..XXXXX....XX.X] duration: 30.00136446s FAIL (2025-07-26 23:26:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001389357s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX......X.X.......X.......X........XX....X....X.X..X.....] duration: 16.645526314s FAIL (2025-07-26 23:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.645555496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [.......X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 30.107610768s FAIL (2025-07-26 23:26:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.107636487s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..........................X......XXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.107225743s FAIL (2025-07-26 23:25:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== piece_id=RP4N7COFERCBSOEHZBQXUQMMZTDPCGEKYOZOBWHC64EGOPOU2SJA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.521556811 (59h49m53.711855397s) [noise=true , address=104.36.82.13:4185 , piece_id=XY745OFR3S6DM5WEPSOYT6XQOD7UZODXCKLQHGLGN2DV5TZ7E56Q, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.521694903 (59h49m53.711993482s) [noise=true , address=104.36.82.4:4801 , piece_id=BJI7K23MY43ZU4FRNQ6353FBDLZJJVCU4QOMIKEYFTC76U25MYCA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.521949672 (59h49m53.712248252s) [noise=true , address=104.36.82.2:4731 , piece_id=MADZ2W2CN4GMKUF3GBC2AAL3SWZDPMGDRARZ5AU7FQZNQVG4B23Q, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.521992670 (59h49m53.712291249s) [noise=true , address=104.36.82.5:4837 , piece_id=E2AYOINJV6MTYC7ZELZKQSTJXKPKW4ZFBJXAIAET5JHFUSOII5RA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.522239621 (59h49m53.712538206s) [noise=true , address=45.91.116.12:4225 , piece_id=SPXPOHXTSGML4WS3PO5J3RJ7R6M3RILV5J2BFQ6MZZCA2AMKTO3A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.522358540 (59h49m53.712657119s) [noise=true , address=104.36.82.14:4223 , piece_id=FXWU5ZSLU3LXFTPRMO7JJQR7QE2L4MT6KD3QR27WCNZW4UYQGGQA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.522476358 (59h49m53.712774937s) [noise=true , address=45.91.116.7:4010 , piece_id=X6UXV3KF6LSSMQIU7GUSX67YZ744OWXDQ7FTRPA6UEKTXMZ3KX2A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.522747349 (59h49m53.713045928s) [noise=true , address=104.36.82.4:4804 , piece_id=MFXNYSQU5JPVHGFL5SMSDC3CKLZD5XETG57UB3XSUYKV6ZW7CI2A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.522767309 (59h49m53.713065896s) [noise=true , address=45.91.116.4:4940 , piece_id=YNEZSPETFRX4WDAHNJV6F77TQS3NNQCEL22QU3KJWVBMWFGKKRRQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.523062794 (59h49m53.71336138s) [noise=true , address=104.36.82.6:4874 , piece_id=XT3R2FAOLOWTDCCPQHJCQTYCGYQQG7YULZNZKG5MDZLSUI2X6LPA, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.523377864 (59h49m53.713676457s) [noise=true , address=45.91.116.8:4048 , piece_id=6J25AYTZVMQ53V7VNZR6IFRZCCPRVN2XECNIVYN62TTWBWQB4D7A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.525580365 (59h49m53.715878947s) [noise=true , address=104.36.82.6:4875 , piece_id=PODJYTOIMKEEVNWWULYFMD7VDB7ZXWKXKGT4ZT65IXRH3YUJ2YKQ, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:22.529395907 (59h49m53.719694478s) [noise=true , address=77.90.31.12:4590 , piece_id=KDOBH5N427MJ6FHS5HSJ7RN4J6D7B564FZRTEVX7BWJIQKXENN3A, seg_pos={0 0}, upload=57377]: Uploading piece... Jul 26 23:25:23.144189777 (59h49m54.334488347s) [upload=57377]: Next returned nil segment Jul 26 23:25:23.144213826 (59h49m54.334512415s) [upload=57377]: Waiting for error group managing segments... Jul 26 23:25:23.278194841 (59h49m54.468493426s) [noise=true , address=45.91.116.8:4048 , piece_id=6J25AYTZVMQ53V7VNZR6IFRZCCPRVN2XECNIVYN62TTWBWQB4D7A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:23.278275340 (59h49m54.468573911s) [noise=true , address=77.90.31.14:4661 , piece_id=AGNQIXMJAOYBW3ZIEENJ6N5WC3VTWOWLQ7O3E6YKT5O3XY76SA4A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:23.282615681 (59h49m54.472914266s) [noise=true , address=45.91.116.7:4010 , piece_id=X6UXV3KF6LSSMQIU7GUSX67YZ744OWXDQ7FTRPA6UEKTXMZ3KX2A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:23.284547412 (59h49m54.47484598s) [noise=true , address=45.91.116.12:4225 , piece_id=SPXPOHXTSGML4WS3PO5J3RJ7R6M3RILV5J2BFQ6MZZCA2AMKTO3A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:23.294182510 (59h49m54.484481088s) [noise=true , address=45.91.116.4:4940 , piece_id=YNEZSPETFRX4WDAHNJV6F77TQS3NNQCEL22QU3KJWVBMWFGKKRRQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:24.617608347 (59h49m55.807906912s) [noise=true , address=77.90.31.12:4590 , piece_id=KDOBH5N427MJ6FHS5HSJ7RN4J6D7B564FZRTEVX7BWJIQKXENN3A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:28.022961665 (59h49m59.213260238s) [noise=true , address=104.36.82.12:4149 , piece_id=RP4N7COFERCBSOEHZBQXUQMMZTDPCGEKYOZOBWHC64EGOPOU2SJA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:28.640154174 (59h49m59.83045276s) [noise=true , address=104.36.82.2:4731 , piece_id=MADZ2W2CN4GMKUF3GBC2AAL3SWZDPMGDRARZ5AU7FQZNQVG4B23Q, seg_pos={0 0}, upload=57377]: Done uploading piece. err: <nil> Jul 26 23:25:52.128603704 (59h50m23.318902272s) [noise=true , address=104.36.82.4:4804 , piece_id=MFXNYSQU5JPVHGFL5SMSDC3CKLZD5XETG57UB3XSUYKV6ZW7CI2A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 139.59.144.22:42410->104.36.82.4:4804: use of closed network connection; write tcp 139.59.144.22:42410->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42410->104.36.82.4:4804: use of closed network connection Jul 26 23:25:52.128766118 (59h50m23.31906469s) [noise=true , address=104.36.82.6:4875 , piece_id=PODJYTOIMKEEVNWWULYFMD7VDB7ZXWKXKGT4ZT65IXRH3YUJ2YKQ, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 139.59.144.22:55532->104.36.82.6:4875: use of closed network connection; write tcp 139.59.144.22:55532->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55532->104.36.82.6:4875: use of closed network connection Jul 26 23:25:52.128874956 (59h50m23.319173529s) [noise=true , address=104.36.82.4:4801 , piece_id=BJI7K23MY43ZU4FRNQ6353FBDLZJJVCU4QOMIKEYFTC76U25MYCA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:51056->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:51056->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51056->104.36.82.4:4801: use of closed network connection Jul 26 23:25:52.128956168 (59h50m23.319254748s) [noise=true , address=104.36.82.6:4874 , piece_id=XT3R2FAOLOWTDCCPQHJCQTYCGYQQG7YULZNZKG5MDZLSUI2X6LPA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:44570->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:44570->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44570->104.36.82.6:4874: use of closed network connection Jul 26 23:25:52.129112120 (59h50m23.319410711s) [noise=true , address=104.36.82.7:4911 , piece_id=KJY3JFBFT6WXPDOBJSRDHHIPVIPYNPHEJI4EAJRNAORBD7QGDT4A, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 139.59.144.22:56196->104.36.82.7:4911: use of closed network connection; write tcp 139.59.144.22:56196->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56196->104.36.82.7:4911: use of closed network connection Jul 26 23:25:52.129353724 (59h50m23.319652294s) [noise=true , address=104.36.82.14:4223 , piece_id=FXWU5ZSLU3LXFTPRMO7JJQR7QE2L4MT6KD3QR27WCNZW4UYQGGQA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:52372->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:52372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52372->104.36.82.14:4223: use of closed network connection Jul 26 23:25:52.129390881 (59h50m23.319689455s) [noise=true , address=104.36.82.13:4185 , piece_id=XY745OFR3S6DM5WEPSOYT6XQOD7UZODXCKLQHGLGN2DV5TZ7E56Q, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:47768->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:47768->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47768->104.36.82.13:4185: use of closed network connection Jul 26 23:25:52.129478706 (59h50m23.319777275s) [noise=true , address=104.36.82.5:4837 , piece_id=E2AYOINJV6MTYC7ZELZKQSTJXKPKW4ZFBJXAIAET5JHFUSOII5RA, seg_pos={0 0}, upload=57377]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:57476->104.36.82.5:4837: use of closed network connection; write tcp 139.59.144.22:57476->104.36.82.5:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57476->104.36.82.5:4837: use of closed network connection Jul 26 23:25:52.129510047 (59h50m23.319808629s) [seg_pos={0 0}, upload=57377]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:25:52.233953326 (59h50m23.424251898s) [upload=57377]: Done uploading segments... Jul 26 23:25:52.233988709 (59h50m23.424287289s) [upload=57377]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.107254536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [X.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X...] duration: 39.452826911s FAIL (2025-07-26 23:25:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:15.489277987 (59h46m46.679576569s) [upload=57325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:15.702839669 (59h46m46.893138255s) [seg_pos={0 0}, upload=57325]: Begin upload segment... Jul 26 23:22:15.702913804 (59h46m46.893212391s) [seg_pos={0 0}, upload=57325]: Done begin upload segment. Jul 26 23:22:15.702962240 (59h46m46.893260825s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703624299 (59h46m46.893922886s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703705288 (59h46m46.89400388s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704717183 (59h46m46.895015774s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704911246 (59h46m46.895209836s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705058351 (59h46m46.895356933s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705963686 (59h46m46.896262277s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706013141 (59h46m46.896311734s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706883998 (59h46m46.897182585s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707155160 (59h46m46.89745375s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707530555 (59h46m46.897829145s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708053028 (59h46m46.898351621s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708475199 (59h46m46.89877379s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709329053 (59h46m46.899627645s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709728548 (59h46m46.900027137s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710158627 (59h46m46.900457217s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710555627 (59h46m46.900854216s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710989594 (59h46m46.901288183s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:16.267925926 (59h46m47.458224496s) [upload=57325]: Next returned nil segment Jul 26 23:22:16.267944613 (59h46m47.458243195s) [upload=57325]: Waiting for error group managing segments... Jul 26 23:22:16.480904398 (59h46m47.671202983s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.532367572 (59h46m47.72266619s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.558052644 (59h46m47.748351223s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.945385561 (59h46m48.13568414s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.179396266 (59h46m48.36969484s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.301285788 (59h46m48.491584368s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.602033920 (59h46m48.7923325s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:20.039346202 (59h46m51.229644789s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:32.674372421 (59h47m3.864670986s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:34.394451336 (59h47m5.584749909s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:39.514179337 (59h47m10.704477914s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:51.857893010 (59h47m23.048191575s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:23:15.210646325 (59h47m46.400944904s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210780376 (59h47m46.401078957s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210936257 (59h47m46.401234841s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: expected piece hash; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; EOF Jul 26 23:23:15.211100863 (59h47m46.401399445s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.211326802 (59h47m46.401625387s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection Jul 26 23:23:15.211574205 (59h47m46.401872789s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:15.211611872 (59h47m46.401910454s) [seg_pos={0 0}, upload=57325]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:23:15.320657081 (59h47m46.510955693s) [upload=57325]: Done uploading segments... Jul 26 23:23:15.320691301 (59h47m46.510989891s) [upload=57325]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 39.452860644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..................................X.XX...XX..X..XXXXX....XX.] duration: 29.203261197s FAIL (2025-07-26 23:25:49 UTC) orbiter warning: context deadline exceeded Duration: 29.203469552s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [........X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 30.113789508s FAIL (2025-07-26 23:25:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.113805871s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX......X.X.......X.......X........XX....X....X.X..X....] duration: 51.415482153s FAIL (2025-07-26 23:25:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.415499744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................................X.XX...XX..X..XXXXX....XX] duration: 30.000901962s FAIL (2025-07-26 23:25:20 UTC) orbiter warning: context deadline exceeded Duration: 30.000920834s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...........................X......XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 24.476361957s FAIL (2025-07-26 23:25:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 54.028525113s) [upload=57357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:24:22.838235035 (59h48m54.028533614s) [upload=57357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:24:22.945663729 (59h48m54.135962308s) [seg_pos={0 0}, upload=57357]: Begin upload segment... Jul 26 23:24:22.945735839 (59h48m54.136034419s) [seg_pos={0 0}, upload=57357]: Done begin upload segment. Jul 26 23:24:22.945763072 (59h48m54.136061653s) [noise=true , address=104.36.82.6:4874 , piece_id=ZIIGWCLJSKHIPRQ735LP73LTGC2TTE24CATT5VN2QODILX5XXBPA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.945789868 (59h48m54.136088449s) [noise=true , address=104.36.82.7:4912 , piece_id=Q65ZNY52WI4FWUNJHCVTBFEUBPU5X63FO4EACVH2KWXWBNAOYY5Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946244496 (59h48m54.136543077s) [noise=true , address=104.36.82.10:4078 , piece_id=EHSXAIAOWITJ453WBX26VWDV32GHR5GNOMW4OYKE3GT35I6TQY4Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946360689 (59h48m54.136659278s) [noise=true , address=77.90.31.14:4663 , piece_id=ADF2RXK62KZESRL3U77OO4RFFKKZ7IDGKGKZTZ3RBB2GDKSFZ7ZA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946618388 (59h48m54.136916973s) [noise=true , address=77.90.31.15:4697 , piece_id=FJOY2ORPLDFCMBWCL5E7AMUQPERDRMF4FBZB7POLWTHNRBHE2RBQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946886683 (59h48m54.137185262s) [noise=true , address=104.36.82.5:4838 , piece_id=XLC4ZILJ4N5VTFUAP4L6EDDSFWRBKTDKCSYMPMIWLVSJSJWDAD5A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946958709 (59h48m54.137257298s) [noise=true , address=104.36.82.3:4767 , piece_id=US3LWTN3CCKXMRUV44W4YEN3EZDYNEHZLW3WKR6BDSKQY5OU5ULA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947265483 (59h48m54.137564064s) [noise=true , address=104.36.82.15:4259 , piece_id=AVTVSEGMUHN6KRS6HKIFXWGQZTLM7A4RIIK2FWZFMWKMZMHHJ2MA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947352476 (59h48m54.137651058s) [noise=true , address=45.91.116.8:4047 , piece_id=T62E3SXYUA5JLOWO5SIMYOYPDJSK4UNZZ6MXM6SZVMKK25ULJGQA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947578418 (59h48m54.137877s) [noise=true , address=45.91.116.13:4155 , piece_id=JQYTGBLZQO37FHEEIJMEDOJDQVOVPTDEBT2GD3BPJPHONNQFAGOQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947885820 (59h48m54.138184399s) [noise=true , address=45.91.116.4:4940 , piece_id=MLO5KA7HXZANI5DSBXNCABILCGBKAN3YAOTJCUJBFZDLDC6IUHFQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947989699 (59h48m54.138288383s) [noise=true , address=104.36.82.13:4187 , piece_id=BLH3IV7BVRYFEUSC6AD6W4QGMWHHN27NWMGSFFWNQDKJDDKPBSWQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.948157558 (59h48m54.13845613s) [noise=true , address=104.36.82.6:4873 , piece_id=GF67X3V6EZ2VZLVRDWJKIOYVDV2WUUYML5H3W2ZFZR4NVF7ECEMA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.948924053 (59h48m54.139222638s) [noise=true , address=104.36.82.4:4801 , piece_id=GWIM3PGQCPXXXNPGFZCARY7INDAOJKMPAEFECZJD35RRNHISKU3Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.949519183 (59h48m54.139817764s) [noise=true , address=45.91.116.12:4225 , piece_id=AUMGTQQFUQOKF6OJOOGIJNXHGFIYPSYRKQ32NYG5QZILCTOCIC7A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.950157410 (59h48m54.140455983s) [noise=true , address=45.91.116.7:4009 , piece_id=EV6UFRGREUJZDWZCDVK2YBVRTK52DXC37PFRLCSQEU36POTTP7FA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:23.558378711 (59h48m54.74867729s) [upload=57357]: Next returned nil segment Jul 26 23:24:23.558400678 (59h48m54.748699268s) [upload=57357]: Waiting for error group managing segments... Jul 26 23:24:23.691980225 (59h48m54.882278794s) [noise=true , address=45.91.116.7:4009 , piece_id=EV6UFRGREUJZDWZCDVK2YBVRTK52DXC37PFRLCSQEU36POTTP7FA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.693995260 (59h48m54.884293834s) [noise=true , address=45.91.116.13:4155 , piece_id=JQYTGBLZQO37FHEEIJMEDOJDQVOVPTDEBT2GD3BPJPHONNQFAGOQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.694168969 (59h48m54.884467552s) [noise=true , address=45.91.116.12:4225 , piece_id=AUMGTQQFUQOKF6OJOOGIJNXHGFIYPSYRKQ32NYG5QZILCTOCIC7A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.703799421 (59h48m54.894097993s) [noise=true , address=77.90.31.15:4697 , piece_id=FJOY2ORPLDFCMBWCL5E7AMUQPERDRMF4FBZB7POLWTHNRBHE2RBQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.711956356 (59h48m54.902254933s) [noise=true , address=45.91.116.8:4047 , piece_id=T62E3SXYUA5JLOWO5SIMYOYPDJSK4UNZZ6MXM6SZVMKK25ULJGQA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.716900703 (59h48m54.907199268s) [noise=true , address=45.91.116.4:4940 , piece_id=MLO5KA7HXZANI5DSBXNCABILCGBKAN3YAOTJCUJBFZDLDC6IUHFQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.797443067 (59h48m54.987741638s) [noise=true , address=77.90.31.14:4663 , piece_id=ADF2RXK62KZESRL3U77OO4RFFKKZ7IDGKGKZTZ3RBB2GDKSFZ7ZA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:46.253154173 (59h49m17.443452759s) [noise=true , address=104.36.82.3:4767 , piece_id=US3LWTN3CCKXMRUV44W4YEN3EZDYNEHZLW3WKR6BDSKQY5OU5ULA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:52.570208762 (59h49m23.760507352s) [noise=true , address=104.36.82.6:4874 , piece_id=ZIIGWCLJSKHIPRQ735LP73LTGC2TTE24CATT5VN2QODILX5XXBPA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570281865 (59h49m23.760580446s) [noise=true , address=104.36.82.4:4801 , piece_id=GWIM3PGQCPXXXNPGFZCARY7INDAOJKMPAEFECZJD35RRNHISKU3Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570361788 (59h49m23.760660385s) [noise=true , address=104.36.82.15:4259 , piece_id=AVTVSEGMUHN6KRS6HKIFXWGQZTLM7A4RIIK2FWZFMWKMZMHHJ2MA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570457922 (59h49m23.760756504s) [noise=true , address=104.36.82.10:4078 , piece_id=EHSXAIAOWITJ453WBX26VWDV32GHR5GNOMW4OYKE3GT35I6TQY4Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570636753 (59h49m23.760935336s) [noise=true , address=104.36.82.5:4838 , piece_id=XLC4ZILJ4N5VTFUAP4L6EDDSFWRBKTDKCSYMPMIWLVSJSJWDAD5A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection; write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection Jul 26 23:24:52.570679999 (59h49m23.760978579s) [noise=true , address=104.36.82.13:4187 , piece_id=BLH3IV7BVRYFEUSC6AD6W4QGMWHHN27NWMGSFFWNQDKJDDKPBSWQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection Jul 26 23:24:52.570724372 (59h49m23.761022952s) [noise=true , address=104.36.82.7:4912 , piece_id=Q65ZNY52WI4FWUNJHCVTBFEUBPU5X63FO4EACVH2KWXWBNAOYY5Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection; write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection Jul 26 23:24:52.570760740 (59h49m23.76105932s) [noise=true , address=104.36.82.6:4873 , piece_id=GF67X3V6EZ2VZLVRDWJKIOYVDV2WUUYML5H3W2ZFZR4NVF7ECEMA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection Jul 26 23:24:52.570790389 (59h49m23.761088972s) [seg_pos={0 0}, upload=57357]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:24:52.686516776 (59h49m23.876815344s) [upload=57357]: Done uploading segments... Jul 26 23:24:52.686550213 (59h49m23.876848782s) [upload=57357]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 24.476391563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 30.004241558s FAIL (2025-07-26 23:25:05 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.00426982s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXXXX.XX..X.XXXXXXXX..X..] duration: 44.368045821s FAIL (2025-07-26 23:24:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:15.489277987 (59h46m46.679576569s) [upload=57325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:15.702839669 (59h46m46.893138255s) [seg_pos={0 0}, upload=57325]: Begin upload segment... Jul 26 23:22:15.702913804 (59h46m46.893212391s) [seg_pos={0 0}, upload=57325]: Done begin upload segment. Jul 26 23:22:15.702962240 (59h46m46.893260825s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703624299 (59h46m46.893922886s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.703705288 (59h46m46.89400388s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704717183 (59h46m46.895015774s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.704911246 (59h46m46.895209836s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705058351 (59h46m46.895356933s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.705963686 (59h46m46.896262277s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706013141 (59h46m46.896311734s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.706883998 (59h46m46.897182585s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707155160 (59h46m46.89745375s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.707530555 (59h46m46.897829145s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708053028 (59h46m46.898351621s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.708475199 (59h46m46.89877379s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709329053 (59h46m46.899627645s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.709728548 (59h46m46.900027137s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710158627 (59h46m46.900457217s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710555627 (59h46m46.900854216s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:15.710989594 (59h46m46.901288183s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Uploading piece... Jul 26 23:22:16.267925926 (59h46m47.458224496s) [upload=57325]: Next returned nil segment Jul 26 23:22:16.267944613 (59h46m47.458243195s) [upload=57325]: Waiting for error group managing segments... Jul 26 23:22:16.480904398 (59h46m47.671202983s) [noise=true , address=77.90.31.12:4600 , piece_id=5H6XCHUJNZTZN5FUAPMOK3NUBQGPVSIC2SK24YQ4X6MKKML7F42Q, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.532367572 (59h46m47.72266619s) [noise=true , address=79.127.219.38:4001 , piece_id=HPA36LBV4IF4FOSUJGROHRFIHJCFWMUODXIDWGM6AEKW7R2RIILA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.558052644 (59h46m47.748351223s) [noise=true , address=77.90.31.18:4826 , piece_id=XN35IBFQOD6DHS6Z3VFANSDS3QJ5FWKJXFE3PNSAZAA324XYN4EQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:16.945385561 (59h46m48.13568414s) [noise=true , address=79.127.205.245:4224 , piece_id=SYUULVVVT72JZHXN7KO4URVIDNMY25R62TMNJV5YYP5LBB2PEKSQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.179396266 (59h46m48.36969484s) [noise=true , address=79.127.205.235:4692 , piece_id=MOLM5ORSWWUZNBSLGCRAWRCN2PA54PTK5SGDQKYCBO5K3GCABEGQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.301285788 (59h46m48.491584368s) [noise=true , address=79.127.205.234:4613 , piece_id=IOI63KH37FWKMMJ2XSHCQKR6VHLVGX3NUUUD6THAOQHWA6U7OYHQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:17.602033920 (59h46m48.7923325s) [noise=true , address=45.91.116.9:4277 , piece_id=NHIOJYK2OXWG2GRWCGTGZT3QQRVONBNMHR3YN6CIE5RYL3ABDOLA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:20.039346202 (59h46m51.229644789s) [noise=true , address=64.49.13.140:4508 , piece_id=5VSBZOYWUGYXHIT26NP54VF3GUQRVGROJ3FJJGD56J3Q33FTF7TA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:32.674372421 (59h47m3.864670986s) [noise=true , address=104.36.82.56:4236 , piece_id=AVFPPNXKBI7XHO2B3Z2QLTULBYDJM5ARNZYGC45N4VC33HVEUW2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:34.394451336 (59h47m5.584749909s) [noise=true , address=104.36.82.54:4165 , piece_id=S73PZMXTDUK3RRGTHZQ2M7MHCD47LHFRG4EI2VUXE5XMCTS46ZOQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:39.514179337 (59h47m10.704477914s) [noise=true , address=104.36.82.52:4102 , piece_id=H3K7A6BWMT4ZMPB6V7ZMBYOH3QB6LAYRCNYKG7S3DSXFVDCLAZCQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:22:51.857893010 (59h47m23.048191575s) [noise=true , address=79.127.201.215:4354 , piece_id=BP2AQLVSQ357QFMFPWYOZPO3NEBLHD2EHIZ7KTCLDUJRPNKSP2XQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: <nil> Jul 26 23:23:15.210646325 (59h47m46.400944904s) [noise=true , address=104.36.82.47:4533 , piece_id=6HX5AHB6XVBVCVABIRHAOD3PDTC7BJCQ6XTS7PJI5RNYY4DC2XVQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210780376 (59h47m46.401078957s) [noise=true , address=104.36.82.46:4485 , piece_id=SMT54U2XZUFSFUMOOFWYVL5LS3ZEI5Z62VSNCZBRRXM2SJ6PRC2A, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.210936257 (59h47m46.401234841s) [noise=true , address=104.36.82.42:4813 , piece_id=GL3N4FREFUDUT66OVO4D23SOP3Z3DXNO6PMLLEC3RCWGKGOKMFGA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: expected piece hash; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; write tcp 139.59.144.22:45162->104.36.82.42:4813: use of closed network connection; EOF Jul 26 23:23:15.211100863 (59h47m46.401399445s) [noise=true , address=104.36.82.55:4195 , piece_id=JZSEG3WKYOVWQHSPLMHMO5YAVKEW5RFO7SG32YP2IK7AGIZEQ6GQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:23:15.211326802 (59h47m46.401625387s) [noise=true , address=104.36.82.53:4135 , piece_id=5ZRZWROJ4SZPVCL6GAKXRE67TYDFDUBXWAZB2LNXZSZG26USI2NA, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35150->104.36.82.53:4135: use of closed network connection Jul 26 23:23:15.211574205 (59h47m46.401872789s) [noise=true , address=77.90.31.16:4759 , piece_id=VCC26LRE32OKKPSADBEWOTD4CJ4VFCAFC46AUWYLZKXWDOCYGTQQ, seg_pos={0 0}, upload=57325]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:15.211611872 (59h47m46.401910454s) [seg_pos={0 0}, upload=57325]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:23:15.320657081 (59h47m46.510955693s) [upload=57325]: Done uploading segments... Jul 26 23:23:15.320691301 (59h47m46.510989891s) [upload=57325]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 44.368099185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [............................X......XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.118711514s FAIL (2025-07-26 23:24:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 54.028525113s) [upload=57357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:24:22.838235035 (59h48m54.028533614s) [upload=57357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:24:22.945663729 (59h48m54.135962308s) [seg_pos={0 0}, upload=57357]: Begin upload segment... Jul 26 23:24:22.945735839 (59h48m54.136034419s) [seg_pos={0 0}, upload=57357]: Done begin upload segment. Jul 26 23:24:22.945763072 (59h48m54.136061653s) [noise=true , address=104.36.82.6:4874 , piece_id=ZIIGWCLJSKHIPRQ735LP73LTGC2TTE24CATT5VN2QODILX5XXBPA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.945789868 (59h48m54.136088449s) [noise=true , address=104.36.82.7:4912 , piece_id=Q65ZNY52WI4FWUNJHCVTBFEUBPU5X63FO4EACVH2KWXWBNAOYY5Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946244496 (59h48m54.136543077s) [noise=true , address=104.36.82.10:4078 , piece_id=EHSXAIAOWITJ453WBX26VWDV32GHR5GNOMW4OYKE3GT35I6TQY4Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946360689 (59h48m54.136659278s) [noise=true , address=77.90.31.14:4663 , piece_id=ADF2RXK62KZESRL3U77OO4RFFKKZ7IDGKGKZTZ3RBB2GDKSFZ7ZA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946618388 (59h48m54.136916973s) [noise=true , address=77.90.31.15:4697 , piece_id=FJOY2ORPLDFCMBWCL5E7AMUQPERDRMF4FBZB7POLWTHNRBHE2RBQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946886683 (59h48m54.137185262s) [noise=true , address=104.36.82.5:4838 , piece_id=XLC4ZILJ4N5VTFUAP4L6EDDSFWRBKTDKCSYMPMIWLVSJSJWDAD5A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.946958709 (59h48m54.137257298s) [noise=true , address=104.36.82.3:4767 , piece_id=US3LWTN3CCKXMRUV44W4YEN3EZDYNEHZLW3WKR6BDSKQY5OU5ULA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947265483 (59h48m54.137564064s) [noise=true , address=104.36.82.15:4259 , piece_id=AVTVSEGMUHN6KRS6HKIFXWGQZTLM7A4RIIK2FWZFMWKMZMHHJ2MA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947352476 (59h48m54.137651058s) [noise=true , address=45.91.116.8:4047 , piece_id=T62E3SXYUA5JLOWO5SIMYOYPDJSK4UNZZ6MXM6SZVMKK25ULJGQA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947578418 (59h48m54.137877s) [noise=true , address=45.91.116.13:4155 , piece_id=JQYTGBLZQO37FHEEIJMEDOJDQVOVPTDEBT2GD3BPJPHONNQFAGOQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947885820 (59h48m54.138184399s) [noise=true , address=45.91.116.4:4940 , piece_id=MLO5KA7HXZANI5DSBXNCABILCGBKAN3YAOTJCUJBFZDLDC6IUHFQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.947989699 (59h48m54.138288383s) [noise=true , address=104.36.82.13:4187 , piece_id=BLH3IV7BVRYFEUSC6AD6W4QGMWHHN27NWMGSFFWNQDKJDDKPBSWQ, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.948157558 (59h48m54.13845613s) [noise=true , address=104.36.82.6:4873 , piece_id=GF67X3V6EZ2VZLVRDWJKIOYVDV2WUUYML5H3W2ZFZR4NVF7ECEMA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.948924053 (59h48m54.139222638s) [noise=true , address=104.36.82.4:4801 , piece_id=GWIM3PGQCPXXXNPGFZCARY7INDAOJKMPAEFECZJD35RRNHISKU3Q, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.949519183 (59h48m54.139817764s) [noise=true , address=45.91.116.12:4225 , piece_id=AUMGTQQFUQOKF6OJOOGIJNXHGFIYPSYRKQ32NYG5QZILCTOCIC7A, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:22.950157410 (59h48m54.140455983s) [noise=true , address=45.91.116.7:4009 , piece_id=EV6UFRGREUJZDWZCDVK2YBVRTK52DXC37PFRLCSQEU36POTTP7FA, seg_pos={0 0}, upload=57357]: Uploading piece... Jul 26 23:24:23.558378711 (59h48m54.74867729s) [upload=57357]: Next returned nil segment Jul 26 23:24:23.558400678 (59h48m54.748699268s) [upload=57357]: Waiting for error group managing segments... Jul 26 23:24:23.691980225 (59h48m54.882278794s) [noise=true , address=45.91.116.7:4009 , piece_id=EV6UFRGREUJZDWZCDVK2YBVRTK52DXC37PFRLCSQEU36POTTP7FA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.693995260 (59h48m54.884293834s) [noise=true , address=45.91.116.13:4155 , piece_id=JQYTGBLZQO37FHEEIJMEDOJDQVOVPTDEBT2GD3BPJPHONNQFAGOQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.694168969 (59h48m54.884467552s) [noise=true , address=45.91.116.12:4225 , piece_id=AUMGTQQFUQOKF6OJOOGIJNXHGFIYPSYRKQ32NYG5QZILCTOCIC7A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.703799421 (59h48m54.894097993s) [noise=true , address=77.90.31.15:4697 , piece_id=FJOY2ORPLDFCMBWCL5E7AMUQPERDRMF4FBZB7POLWTHNRBHE2RBQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.711956356 (59h48m54.902254933s) [noise=true , address=45.91.116.8:4047 , piece_id=T62E3SXYUA5JLOWO5SIMYOYPDJSK4UNZZ6MXM6SZVMKK25ULJGQA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.716900703 (59h48m54.907199268s) [noise=true , address=45.91.116.4:4940 , piece_id=MLO5KA7HXZANI5DSBXNCABILCGBKAN3YAOTJCUJBFZDLDC6IUHFQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:23.797443067 (59h48m54.987741638s) [noise=true , address=77.90.31.14:4663 , piece_id=ADF2RXK62KZESRL3U77OO4RFFKKZ7IDGKGKZTZ3RBB2GDKSFZ7ZA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:46.253154173 (59h49m17.443452759s) [noise=true , address=104.36.82.3:4767 , piece_id=US3LWTN3CCKXMRUV44W4YEN3EZDYNEHZLW3WKR6BDSKQY5OU5ULA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: <nil> Jul 26 23:24:52.570208762 (59h49m23.760507352s) [noise=true , address=104.36.82.6:4874 , piece_id=ZIIGWCLJSKHIPRQ735LP73LTGC2TTE24CATT5VN2QODILX5XXBPA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570281865 (59h49m23.760580446s) [noise=true , address=104.36.82.4:4801 , piece_id=GWIM3PGQCPXXXNPGFZCARY7INDAOJKMPAEFECZJD35RRNHISKU3Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570361788 (59h49m23.760660385s) [noise=true , address=104.36.82.15:4259 , piece_id=AVTVSEGMUHN6KRS6HKIFXWGQZTLM7A4RIIK2FWZFMWKMZMHHJ2MA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570457922 (59h49m23.760756504s) [noise=true , address=104.36.82.10:4078 , piece_id=EHSXAIAOWITJ453WBX26VWDV32GHR5GNOMW4OYKE3GT35I6TQY4Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:52.570636753 (59h49m23.760935336s) [noise=true , address=104.36.82.5:4838 , piece_id=XLC4ZILJ4N5VTFUAP4L6EDDSFWRBKTDKCSYMPMIWLVSJSJWDAD5A, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection; write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36716->104.36.82.5:4838: use of closed network connection Jul 26 23:24:52.570679999 (59h49m23.760978579s) [noise=true , address=104.36.82.13:4187 , piece_id=BLH3IV7BVRYFEUSC6AD6W4QGMWHHN27NWMGSFFWNQDKJDDKPBSWQ, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41568->104.36.82.13:4187: use of closed network connection Jul 26 23:24:52.570724372 (59h49m23.761022952s) [noise=true , address=104.36.82.7:4912 , piece_id=Q65ZNY52WI4FWUNJHCVTBFEUBPU5X63FO4EACVH2KWXWBNAOYY5Q, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection; write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42532->104.36.82.7:4912: use of closed network connection Jul 26 23:24:52.570760740 (59h49m23.76105932s) [noise=true , address=104.36.82.6:4873 , piece_id=GF67X3V6EZ2VZLVRDWJKIOYVDV2WUUYML5H3W2ZFZR4NVF7ECEMA, seg_pos={0 0}, upload=57357]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46050->104.36.82.6:4873: use of closed network connection Jul 26 23:24:52.570790389 (59h49m23.761088972s) [seg_pos={0 0}, upload=57357]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:24:52.686516776 (59h49m23.876815344s) [upload=57357]: Done uploading segments... Jul 26 23:24:52.686550213 (59h49m23.876848782s) [upload=57357]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.118741266s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....................................X.XX...XX..X..XXXXX....X] duration: 30.001136804s FAIL (2025-07-26 23:24:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001163054s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..X.XX......X.X.......X.......X........XX....X....X.X..X...] duration: 33.478459806s FAIL (2025-07-26 23:24:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.478488023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [..........X..................X.....XXXXXX.XXXXXXXXXXXXXXXXXX] duration: 30.112570964s FAIL (2025-07-26 23:24:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.112599621s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.............................X......XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.106352465s FAIL (2025-07-26 23:24:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:45.056689365 (59h48m16.246987951s) [seg_pos={0 0}, upload=57337]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:23:45.194342480 (59h48m16.384641061s) [upload=57337]: Done uploading segments... Jul 26 23:23:45.194376076 (59h48m16.384674662s) [upload=57337]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:23:45.626800836 (59h48m16.817099415s) [upload=57346]: Starting upload Jul 26 23:23:45.626869246 (59h48m16.817167828s) [upload=57346]: Done starting upload Jul 26 23:23:45.627186961 (59h48m16.817485533s) [upload=57346]: Uploading segments... Jul 26 23:23:45.627255956 (59h48m16.817554524s) [upload=57346]: Got next segment. Inline: false Jul 26 23:23:45.627297949 (59h48m16.817596527s) [upload=57346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:23:45.627303545 (59h48m16.817602125s) [upload=57346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:23:45.735593819 (59h48m16.925892398s) [seg_pos={0 0}, upload=57346]: Begin upload segment... Jul 26 23:23:45.735639846 (59h48m16.925938419s) [seg_pos={0 0}, upload=57346]: Done begin upload segment. Jul 26 23:23:45.735678370 (59h48m16.925976955s) [noise=true , address=45.91.116.12:4225 , piece_id=EEDPTXVR4R5VWS625THCOEIMCAIPVWDWDDKZOQUVUO6N5NPREHEQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.735761456 (59h48m16.926060036s) [noise=true , address=45.91.116.4:4939 , piece_id=5SDTD7YM56KLRHEHQDN6E744BVSFMFT7UI3JDKHSLY7VMI3ZLBKA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.735987535 (59h48m16.926286123s) [noise=true , address=45.91.116.11:4118 , piece_id=U33LP34EWXXUW7YMUJYYFQURUEDOLNO5REI3MUYRSFD34HXNK6PA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.736214418 (59h48m16.926512997s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTZIAJADOEJLV2237ET2VRC4L5USXGIUQ5B26VZQWEXSBSR5FPQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.736513722 (59h48m16.926812311s) [noise=true , address=104.36.82.16:4293 , piece_id=HTILTI5O4KCVVKIHJEXRV7VSQXUJU74G7FYNHDBBP6WTR4TRDGFQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.736788461 (59h48m16.927087052s) [noise=true , address=45.91.116.9:4261 , piece_id=TWRDKOM3VSBF7RBWLXX7AGZFRLSXDJGKMWLNGBO2MLCDUSDRLMKA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.736863338 (59h48m16.927161919s) [noise=true , address=45.91.116.10:4083 , piece_id=AJFKC6B4LYAVCXGKJBKG3EKLGN5BCY7NH6NMEJZ33ELK53U22XWQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737121272 (59h48m16.927419854s) [noise=true , address=45.91.116.6:4974 , piece_id=263UQRR6Z57UBTKW5B4XQMJFHABOKICEHXIGJPDAAUV5HDQOVAMQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737206180 (59h48m16.927504762s) [noise=true , address=104.36.82.3:4767 , piece_id=HEIAWUA2YJKSHJPQX5MNZ5BHBZGPF743M56LPD4PK2I6T6JBWJCQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737408063 (59h48m16.927706653s) [noise=true , address=45.91.116.7:4011 , piece_id=HJJVLC32MGNBJ55ICH2ZZUDW6N6X5SVUECMYFQZB73D6LCYLXEAA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737521196 (59h48m16.927819775s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZKOJG4A3W2FAQCE575SWEPDL53FS262NJYMQVNL5UT5PWXUHWBQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737798289 (59h48m16.92809687s) [noise=true , address=77.90.31.14:4661 , piece_id=7UNDYZZGBVC3IKM3BJH35QODRPCJTYKLTNJKHZ6AT6ZG73BQ3BEA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.737839390 (59h48m16.92813797s) [noise=true , address=45.91.116.8:4045 , piece_id=RDVEC372W77XJHEYWW6ADHZYFCGI773JMX6A42DBNM7BUTVOAJJQ, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.738111417 (59h48m16.928409997s) [noise=true , address=104.36.82.8:4971 , piece_id=7E47D46DFVWDOVMLUW624PSYTRLXMJ7WVIUT6CU3G2VTYBFTVJUA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.738404378 (59h48m16.928702959s) [noise=true , address=77.90.31.10:4519 , piece_id=BILEUKFCHXXIQYQEWNU4NMGHTTTZZXX7UWCGEWEXYWUNNJJMJCAA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:45.738687304 (59h48m16.928985886s) [noise=true , address=104.36.82.8:4972 , piece_id=TG7DOPYFRSTY5D342Q6F2CXLDRFAS7R7X3KSJZOIQII6WRZF2BRA, seg_pos={0 0}, upload=57346]: Uploading piece... Jul 26 23:23:46.344882164 (59h48m17.535180729s) [upload=57346]: Next returned nil segment Jul 26 23:23:46.344903141 (59h48m17.535201723s) [upload=57346]: Waiting for error group managing segments... Jul 26 23:23:46.484064953 (59h48m17.674363533s) [noise=true , address=45.91.116.12:4225 , piece_id=EEDPTXVR4R5VWS625THCOEIMCAIPVWDWDDKZOQUVUO6N5NPREHEQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.486506151 (59h48m17.676804716s) [noise=true , address=45.91.116.4:4939 , piece_id=5SDTD7YM56KLRHEHQDN6E744BVSFMFT7UI3JDKHSLY7VMI3ZLBKA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.487146328 (59h48m17.677444897s) [noise=true , address=45.91.116.10:4083 , piece_id=AJFKC6B4LYAVCXGKJBKG3EKLGN5BCY7NH6NMEJZ33ELK53U22XWQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.488169054 (59h48m17.678467626s) [noise=true , address=45.91.116.8:4045 , piece_id=RDVEC372W77XJHEYWW6ADHZYFCGI773JMX6A42DBNM7BUTVOAJJQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.491747223 (59h48m17.682045803s) [noise=true , address=45.91.116.7:4011 , piece_id=HJJVLC32MGNBJ55ICH2ZZUDW6N6X5SVUECMYFQZB73D6LCYLXEAA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.498632427 (59h48m17.688930997s) [noise=true , address=77.90.31.14:4661 , piece_id=7UNDYZZGBVC3IKM3BJH35QODRPCJTYKLTNJKHZ6AT6ZG73BQ3BEA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.501416265 (59h48m17.691714844s) [noise=true , address=45.91.116.6:4974 , piece_id=263UQRR6Z57UBTKW5B4XQMJFHABOKICEHXIGJPDAAUV5HDQOVAMQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.552035992 (59h48m17.742334577s) [noise=true , address=45.91.116.11:4118 , piece_id=U33LP34EWXXUW7YMUJYYFQURUEDOLNO5REI3MUYRSFD34HXNK6PA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.570147816 (59h48m17.760446395s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTZIAJADOEJLV2237ET2VRC4L5USXGIUQ5B26VZQWEXSBSR5FPQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:46.956853687 (59h48m18.147152269s) [noise=true , address=45.91.116.9:4261 , piece_id=TWRDKOM3VSBF7RBWLXX7AGZFRLSXDJGKMWLNGBO2MLCDUSDRLMKA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:23:48.518769382 (59h48m19.709067973s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZKOJG4A3W2FAQCE575SWEPDL53FS262NJYMQVNL5UT5PWXUHWBQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:24:06.573457842 (59h48m37.763756415s) [noise=true , address=104.36.82.3:4767 , piece_id=HEIAWUA2YJKSHJPQX5MNZ5BHBZGPF743M56LPD4PK2I6T6JBWJCQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: <nil> Jul 26 23:24:15.356475674 (59h48m46.546774257s) [noise=true , address=104.36.82.16:4293 , piece_id=HTILTI5O4KCVVKIHJEXRV7VSQXUJU74G7FYNHDBBP6WTR4TRDGFQ, seg_pos={0 0}, upload=57346]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:33084->104.36.82.16:4293: use of closed network connection; write tcp 139.59.144.22:33084->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33084->104.36.82.16:4293: use of closed network connection Jul 26 23:24:15.356606363 (59h48m46.546904942s) [noise=true , address=104.36.82.8:4971 , piece_id=7E47D46DFVWDOVMLUW624PSYTRLXMJ7WVIUT6CU3G2VTYBFTVJUA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52274->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:52274->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52274->104.36.82.8:4971: use of closed network connection Jul 26 23:24:15.356642310 (59h48m46.546940889s) [noise=true , address=77.90.31.10:4519 , piece_id=BILEUKFCHXXIQYQEWNU4NMGHTTTZZXX7UWCGEWEXYWUNNJJMJCAA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Jul 26 23:24:15.356693590 (59h48m46.54699217s) [noise=true , address=104.36.82.8:4972 , piece_id=TG7DOPYFRSTY5D342Q6F2CXLDRFAS7R7X3KSJZOIQII6WRZF2BRA, seg_pos={0 0}, upload=57346]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:32818->104.36.82.8:4972: use of closed network connection; write tcp 139.59.144.22:32818->104.36.82.8:4972: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32818->104.36.82.8:4972: use of closed network connection Jul 26 23:24:15.356724785 (59h48m46.547023367s) [seg_pos={0 0}, upload=57346]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:24:15.461255028 (59h48m46.6515536s) [upload=57346]: Done uploading segments... Jul 26 23:24:15.461282788 (59h48m46.651581367s) [upload=57346]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.106370452s core uplink us-select-1

1
...
6433
6434
6435
6436
6437
...
6952