Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 170765 reports (Page 3274 of 3416)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X.X.....X.........X.................XX..X.......] duration: 16.186633463s FAIL (2025-07-09 02:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.186668385s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.] duration: 46.035881933s FAIL (2025-07-09 02:26:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.035908234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........................X........X........X...............] duration: 2.671178915s FAIL (2025-07-09 02:26:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.67121142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........................X........X........X..............] duration: 18.777089782s FAIL (2025-07-09 02:25:25 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 18.777193183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X......X.X.....X.........X.................XX..X......] duration: 6.852267853s FAIL (2025-07-09 02:25:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.852297511s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX] duration: 1m0.00169484s FAIL (2025-07-09 02:25:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001740376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X......X.X.....X.........X.................XX..X.....] duration: 21.218269029s FAIL (2025-07-09 02:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.218308423s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........................X........X........X.............] duration: 2.827986892s FAIL (2025-07-09 02:24:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.828020375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...............X.......] duration: 2.564757023s FAIL (2025-07-09 02:24:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 2.564779838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...............X......] duration: 47.10578903s FAIL (2025-07-09 02:24:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 47.105823661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.X] duration: 1m0.000783073s FAIL (2025-07-09 02:24:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000857046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........................X........X........X............] duration: 2.373046624s FAIL (2025-07-09 02:24:08 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.37307772s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X......X.X.....X.........X.................XX..X....] duration: 27.171549498s FAIL (2025-07-09 02:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.171579086s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........................X........X........X...........] duration: 2.322220455s FAIL (2025-07-09 02:23:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.322241557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........................X........X........X..........] duration: 2.298778113s FAIL (2025-07-09 02:23:12 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.298807995s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X......X.X.....X.........X.................XX..X...] duration: 3.326057636s FAIL (2025-07-09 02:23:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.32609389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...............X.....] duration: 10.651254646s FAIL (2025-07-09 02:22:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 10.651302047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...............X....] duration: 37.232717788s FAIL (2025-07-09 02:22:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 37.232737148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.] duration: 21.346070514s FAIL (2025-07-09 02:22:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.346090004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.........................X........X........X.........] duration: 2.867728693s FAIL (2025-07-09 02:22:21 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.867810312s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [X.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX] duration: 1m0.000660186s FAIL (2025-07-09 02:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000682491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.........................X........X........X........] duration: 2.32421389s FAIL (2025-07-09 02:22:01 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.324270523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...............X...] duration: 35.644319468s FAIL (2025-07-09 02:21:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 35.644343948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X......X.X.....X.........X.................XX..X..] duration: 2.66583604s FAIL (2025-07-09 02:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.665853609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.........................X........X........X.......] duration: 2.465563003s FAIL (2025-07-09 02:21:42 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.465594156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X......X.X.....X.........X.................XX..X.] duration: 2.602871333s FAIL (2025-07-09 02:21:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.60290388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.........................X........X........X......] duration: 2.334653543s FAIL (2025-07-09 02:21:15 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.334759118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...............X..] duration: 35.166565937s FAIL (2025-07-09 02:21:04 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 35.16659038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X......X.X.....X.........X.................XX..X] duration: 30.00099883s FAIL (2025-07-09 02:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00102323s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.........................X........X........X.....] duration: 2.743149541s FAIL (2025-07-09 02:20:43 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.743179577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XX] duration: 43.602397901s FAIL (2025-07-09 02:20:37 UTC) orbiter warning: unexpected EOF Duration: 43.602415122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...............X.] duration: 35.298339094s FAIL (2025-07-09 02:20:29 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 35.29837606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.........................X........X........X....] duration: 2.332773044s FAIL (2025-07-09 02:20:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.332808614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..X] duration: 1m0.00091274s FAIL (2025-07-09 02:19:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000929813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...............X] duration: 1m0.003145335s FAIL (2025-07-09 02:19:53 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment... Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment. Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece... Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments... Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil> Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces. Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments... Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil> Duration: 1m0.003179263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X......X.X.....X.........X.................XX..] duration: 2.879242574s FAIL (2025-07-09 02:19:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.879312023s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.............X......X.X.....X.........X.................XX.] duration: 20.313439972s FAIL (2025-07-09 02:19:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.313469115s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.........................X........X........X...] duration: 9.388512072s FAIL (2025-07-09 02:19:29 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 9.388539206s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.............X......X.X.....X.........X.................XX] duration: 30.000647447s FAIL (2025-07-09 02:19:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000663476s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.........................X........X........X..] duration: 2.424348908s FAIL (2025-07-09 02:18:56 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.4243818s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..] duration: 48.94977665s FAIL (2025-07-09 02:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.949814804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.............X......X.X.....X.........X.................X] duration: 30.00164591s FAIL (2025-07-09 02:18:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001671896s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 31.46119216s FAIL (2025-07-09 02:18:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2682347s) [upload=301406]: Got next segment. Inline: false Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment... Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment. Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments... Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments... Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.461241541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [.XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX.] duration: 45.580214804s FAIL (2025-07-09 02:18:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.580231079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 38.487079844s FAIL (2025-07-09 02:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2682347s) [upload=301406]: Got next segment. Inline: false Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment... Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment. Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece... Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments... Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil> Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments... Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.487177392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X.............X......X.X.....X.........X.................] duration: 17.410808519s FAIL (2025-07-09 02:17:53 UTC) orbiter warning: context deadline exceeded Duration: 17.410848151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.........................X........X........X.] duration: 2.749336003s FAIL (2025-07-09 02:17:42 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 2.749363541s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..X.............X......X.X.....X.........X................] duration: 19.427145994s FAIL (2025-07-09 02:17:32 UTC) orbiter warning: context deadline exceeded Duration: 19.427175009s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX] duration: 1m0.001990671s FAIL (2025-07-09 02:17:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002014724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.........................X........X........X] duration: 30.003102949s FAIL (2025-07-09 02:17:05 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31.372731614s) [seg_pos={0 0}, upload=301549]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:16:31.489491468 (377h5m31.372905094s) [upload=301549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:16:31.489497413 (377h5m31.372911039s) [upload=301549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:16:31.643765289 (377h5m31.527178926s) [upload=301549]: Done uploading segments... Jul 9 02:16:31.643789455 (377h5m31.527203095s) [upload=301549]: Upload finished. err: <nil> Jul 9 02:16:35.533078821 (377h5m35.416492448s) [upload=301555]: Starting upload Jul 9 02:16:35.533172714 (377h5m35.416586342s) [upload=301555]: Done starting upload Jul 9 02:16:35.533291187 (377h5m35.416704815s) [upload=301555]: Uploading segments... Jul 9 02:16:35.533568216 (377h5m35.416981832s) [upload=301555]: Got next segment. Inline: false Jul 9 02:16:35.533613399 (377h5m35.417027016s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:16:35.533620376 (377h5m35.417034004s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:16:35.644812582 (377h5m35.528226211s) [seg_pos={0 0}, upload=301555]: Begin upload segment... Jul 9 02:16:35.644985999 (377h5m35.528399617s) [seg_pos={0 0}, upload=301555]: Done begin upload segment. Jul 9 02:16:35.645153902 (377h5m35.528567528s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645257814 (377h5m35.528671442s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645294584 (377h5m35.528708214s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645339787 (377h5m35.528753413s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645371579 (377h5m35.528785205s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645435078 (377h5m35.528848778s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645493421 (377h5m35.528907048s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645541238 (377h5m35.528954867s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645586449 (377h5m35.529000075s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645628337 (377h5m35.529041973s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645685987 (377h5m35.52909962s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645745859 (377h5m35.529159486s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645779748 (377h5m35.529193374s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645820456 (377h5m35.529234082s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645853228 (377h5m35.529266846s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:35.645899802 (377h5m35.529313429s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Uploading piece... Jul 9 02:16:36.254004148 (377h5m36.137417765s) [upload=301555]: Next returned nil segment Jul 9 02:16:36.254025834 (377h5m36.137439472s) [upload=301555]: Waiting for error group managing segments... Jul 9 02:16:36.391118235 (377h5m36.274531861s) [noise=true , address=77.90.31.17:4770 , piece_id=LRO2SWPQLEMCX5YJKBZZWOWBNBK7Z2LOKYPZI6HVNQZTSUFZP7NQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.397843079 (377h5m36.281256695s) [noise=true , address=45.91.116.12:4225 , piece_id=XQTV2BWAHH64N3BMCTRZSOXSRZ4X4XOEJKZE7L7NTNKCXABVYQWQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402231092 (377h5m36.285644718s) [noise=true , address=104.36.82.9:4007 , piece_id=O6G7RDRUAQIHXEW52XMXW3T6VFU75O4G633RNMNMQMOLC3YF43LQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402310074 (377h5m36.285723694s) [noise=true , address=77.90.31.11:4553 , piece_id=7N4TJCENFWZHYTYL4PJDOW66OKFJ7TBRYMCUHAVOWXKUSGUHO2OQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.402967729 (377h5m36.286381346s) [noise=true , address=45.91.116.10:4083 , piece_id=WHFDU5JI3AJIIGDXW3AAGQR6TSU3WMLKZDRJDGSTKI7BOH3BFDNA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.403406566 (377h5m36.286820185s) [noise=true , address=45.91.116.7:4009 , piece_id=OYP57CNT5DEDOYBF3UYKHIVR67EEY4S6RDLVWVT5LDFU2XTNVR4Q, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.405502593 (377h5m36.28891622s) [noise=true , address=45.91.116.7:4010 , piece_id=6MHVEPJA6LWMJGMDPNCUVDQLSDWGFJZTHV4PX5JGYXXITYHXC4UA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.406357723 (377h5m36.289771343s) [noise=true , address=45.91.116.9:4261 , piece_id=IKMZN4GP5COQB4JSTQKW6P7FBQ3KQ5EGDBFTBYDQZZPFMHPWS72A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.408691912 (377h5m36.292105543s) [noise=true , address=45.91.116.13:4153 , piece_id=GQKEMOBKJJZVMY2PLACAYPB6QCXQ6PK3TEOT5PVEK2UPKQYXIW4A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.412343181 (377h5m36.295756809s) [noise=true , address=45.91.116.13:4154 , piece_id=5BU5W7XZI2VJRWSWAPCPUSEY4SP4HMFRBB2ULLCJ4KJNDC2T43AQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.471805671 (377h5m36.35521929s) [noise=true , address=45.91.116.4:4940 , piece_id=MLCY6BWA34XTEXJSDXYQQJUAWVC5SMTQCM57KVGZC7NWA4Q6MB7A, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:16:36.590596314 (377h5m36.474009931s) [noise=true , address=45.91.116.11:4117 , piece_id=VFIH3ZEDEQHOYS53UBWPXRZE5YD7J4SWBJXKUAJN6H3A766OJSCQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804501216 (377h6m4.687914894s) [noise=true , address=104.36.82.2:4731 , piece_id=RC6WSWPX7I4TSNRN4EAHSS5E7VDZVSQ2ZHV52VGSLWMQHXG7BVVQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: <nil> Jul 9 02:17:04.804559634 (377h6m4.687973263s) [seg_pos={0 0}, upload=301555]: Segment reached optimal threshold of 13 pieces. Jul 9 02:17:04.804742356 (377h6m4.688155989s) [noise=true , address=104.36.82.13:4185 , piece_id=ASGR4KJPFGIMU6HWNGFXOP6JDR6NECH4I24PP7SCH3MV4QRJOOPQ, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Jul 9 02:17:04.805081728 (377h6m4.688495353s) [noise=true , address=104.36.82.11:4114 , piece_id=ZL7L2XTELKGO2OPYL2ZLH72BGYSLSLS4KC6E3ZF7IO3ZR5PV7VKA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49892->104.36.82.11:4114: use of closed network connection Jul 9 02:17:04.805159167 (377h6m4.688572781s) [noise=true , address=104.36.82.8:4969 , piece_id=OT5NFCHQUZ2TXKKI6XV6356E5YUD6XTT6T6H5LCIPQR5ZUXLECPA, seg_pos={0 0}, upload=301555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39628->104.36.82.8:4969: use of closed network connection Jul 9 02:17:04.805175806 (377h6m4.68858942s) [seg_pos={0 0}, upload=301555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:17:04.805279601 (377h6m4.688693233s) [upload=301555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:17:04.805285998 (377h6m4.688699616s) [upload=301555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:17:04.963658930 (377h6m4.847072546s) [upload=301555]: Done uploading segments... Jul 9 02:17:04.963683840 (377h6m4.847097453s) [upload=301555]: Upload finished. err: <nil> Duration: 30.003158041s core uplink us-select-1

1
...
3272
3273
3274
3275
3276
...
3416