Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X..X..X.............X......X.X.....X.........X.......] duration:
11.919696286s
FAIL (2025-07-09 02:12:41 UTC)
orbiter warning: context deadline exceeded
Duration: 11.919766515s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X.........................X........X....] duration:
2.571324174s
FAIL (2025-07-09 02:12:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.571375452s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
29.320959238s
FAIL (2025-07-09 02:12:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 29.320987895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [..XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.] duration:
26.149008937s
FAIL (2025-07-09 02:12:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.149023838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X.........................X........X...] duration:
2.315327197s
FAIL (2025-07-09 02:11:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.315382861s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX] duration:
1m0.000544128s
FAIL (2025-07-09 02:11:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000560036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X..X..X.............X......X.X.....X.........X......] duration:
3.788949888s
FAIL (2025-07-09 02:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 3.788972844s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
13.180258461s
FAIL (2025-07-09 02:11:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.180349816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X.........................X........X..] duration:
2.331323088s
FAIL (2025-07-09 02:10:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.33135254s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
17.001192021s
FAIL (2025-07-09 02:10:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.001239219s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X..X..X.............X......X.X.....X.........X.....] duration:
8.708703999s
FAIL (2025-07-09 02:10:35 UTC)
orbiter warning: context deadline exceeded
Duration: 8.708752319s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXX] duration:
1m0.000649012s
FAIL (2025-07-09 02:10:30 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000678253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.........................X........X.] duration:
2.322661174s
FAIL (2025-07-09 02:10:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.322742576s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
34.950431337s
FAIL (2025-07-09 02:10:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 34.950474073s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X.........................X........X] duration:
30.106787447s
FAIL (2025-07-09 02:09:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.106819442s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [.X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXX] duration:
1m0.00131243s
FAIL (2025-07-09 02:09:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00132433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
33.257368338s
FAIL (2025-07-09 02:09:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 33.257398214s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X..X..X.............X......X.X.....X.........X....] duration:
8.570230245s
FAIL (2025-07-09 02:09:23 UTC)
orbiter warning: context deadline exceeded
Duration: 8.570267107s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........X..X..X.............X......X.X.....X.........X...] duration:
11.353360664s
FAIL (2025-07-09 02:08:32 UTC)
orbiter warning: context deadline exceeded
Duration: 11.353391801s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXX] duration:
1m0.00055743s
FAIL (2025-07-09 02:08:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000581934s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
35.487323498s
FAIL (2025-07-09 02:08:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 35.4873681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X.........................X........] duration:
2.360538485s
FAIL (2025-07-09 02:07:53 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.360573196s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............X..X..X.............X......X.X.....X.........X..] duration:
2.962340435s
FAIL (2025-07-09 02:07:47 UTC)
orbiter warning: context deadline exceeded
Duration: 2.962379056s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
46.961947337s
FAIL (2025-07-09 02:07:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 46.961974973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXX] duration:
1m0.001539405s
FAIL (2025-07-09 02:07:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001556419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
45.006731792s
FAIL (2025-07-09 02:06:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 45.006753226s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.........................X.......] duration:
2.288860526s
FAIL (2025-07-09 02:06:51 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.28889956s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............X..X..X.............X......X.X.....X.........X.] duration:
8.97175432s
FAIL (2025-07-09 02:06:41 UTC)
orbiter warning: context deadline exceeded
Duration: 8.972365727s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [.X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXX] duration:
1m0.002144695s
FAIL (2025-07-09 02:06:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.002167009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.104578238s
FAIL (2025-07-09 02:06:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.104602049s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............X..X..X.............X......X.X.....X.........X] duration:
30.00124919s
FAIL (2025-07-09 02:06:10 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00128628s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X.........................X......] duration:
2.286487201s
FAIL (2025-07-09 02:05:38 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.286577529s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
39 out of 60 is failed (threshold=3) [..X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXX] duration:
1m0.001976705s
FAIL (2025-07-09 02:05:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001994723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...............X..X..X.............X......X.X.....X.........] duration:
11.58340568s
FAIL (2025-07-09 02:05:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.583438764s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X.........................X.....] duration:
2.313379818s
FAIL (2025-07-09 02:04:55 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.313412905s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.........................X....] duration:
2.31811926s
FAIL (2025-07-09 02:04:25 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.318154667s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [...X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XX] duration:
1m0.002856888s
FAIL (2025-07-09 02:04:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.003000503s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [................X..X..X.............X......X.X.....X........] duration:
2.849146596s
FAIL (2025-07-09 02:04:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.849172768s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.................X..X..X.............X......X.X.....X.......] duration:
12.469075158s
FAIL (2025-07-09 02:03:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.469089653s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..................X..X..X.............X......X.X.....X......] duration:
22.092286067s
FAIL (2025-07-09 02:03:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.092332522s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.........................X...] duration:
2.437532546s
FAIL (2025-07-09 02:03:36 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.43756981s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
37 out of 60 is failed (threshold=3) [....X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.X] duration:
1m0.001372638s
FAIL (2025-07-09 02:03:21 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001391782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.........................X..] duration:
2.359026365s
FAIL (2025-07-09 02:03:14 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.359068886s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.........................X.] duration:
2.533942584s
FAIL (2025-07-09 02:02:47 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.533972768s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...................X..X..X.............X......X.X.....X.....] duration:
6.642217124s
FAIL (2025-07-09 02:02:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.642243922s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.........................X] duration:
30.008505786s
FAIL (2025-07-09 02:02:44 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 30.008537334s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [....................X..X..X.............X......X.X.....X....] duration:
18.642492745s
FAIL (2025-07-09 02:02:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.642525048s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [.....X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.] duration:
52.22821701s
FAIL (2025-07-09 02:02:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 52.228246881s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.742191106s
FAIL (2025-07-09 02:01:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
UARKKKYWHS6OJARTQ, seg_pos={0 0}, upload=301070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Jul 9 01:40:04.447461354 (376h29m4.330874974s) [seg_pos={0 0}, upload=301070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 01:40:04.447566480 (376h29m4.3309801s) [upload=301070]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:40:04.447573285 (376h29m4.330986904s) [upload=301070]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:40:04.593277678 (376h29m4.476691297s) [upload=301070]: Done uploading segments...
Jul 9 01:40:04.593300848 (376h29m4.476714464s) [upload=301070]: Upload finished. err: <nil>
Jul 9 01:41:14.682414521 (376h30m14.565828152s) [upload=301082]: Starting upload
Jul 9 01:41:14.682503694 (376h30m14.565917327s) [upload=301082]: Done starting upload
Jul 9 01:41:14.682895930 (376h30m14.566309543s) [upload=301082]: Uploading segments...
Jul 9 01:41:14.682966356 (376h30m14.566379999s) [upload=301082]: Got next segment. Inline: false
Jul 9 01:41:14.683011792 (376h30m14.566425428s) [upload=301082]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:14.683017185 (376h30m14.566430818s) [upload=301082]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:14.790212148 (376h30m14.673625784s) [seg_pos={0 0}, upload=301082]: Begin upload segment...
Jul 9 01:41:14.790319542 (376h30m14.673733178s) [seg_pos={0 0}, upload=301082]: Done begin upload segment.
Jul 9 01:41:14.790497270 (376h30m14.673910907s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.790540123 (376h30m14.67395376s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.791164639 (376h30m14.674578277s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.792366139 (376h30m14.675779767s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.793650193 (376h30m14.677064072s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.794895159 (376h30m14.678308796s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.795507503 (376h30m14.678921142s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.796576755 (376h30m14.679990392s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.798463453 (376h30m14.681877094s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.799335495 (376h30m14.682749139s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.800229900 (376h30m14.683643538s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.806816135 (376h30m14.690229772s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.808677646 (376h30m14.692091277s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.810758199 (376h30m14.694171828s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.812672044 (376h30m14.696085673s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:14.814773304 (376h30m14.698186935s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Uploading piece...
Jul 9 01:41:15.406098847 (376h30m15.289512467s) [upload=301082]: Next returned nil segment
Jul 9 01:41:15.406117780 (376h30m15.289531408s) [upload=301082]: Waiting for error group managing segments...
Jul 9 01:41:15.538146641 (376h30m15.42156026s) [noise=true , address=45.91.116.14:4190 , piece_id=FNP5PTK6DKU4TYBYAEF6FZCIKXMOM2PDB32UFF67GWVQ4LQJDRRQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.539592447 (376h30m15.423006066s) [noise=true , address=45.91.116.12:4226 , piece_id=I4TVQXPSTRUQUWN4IQLDFZ3SCRHBCI2ZTB5KKQF4H56RQUEWZXYQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.540032848 (376h30m15.423446465s) [noise=true , address=45.91.116.9:4262 , piece_id=W4G36J5XZFKTAJNB4VBI55ECCSB3ZXHF2SPD3JG7XECUYCWED6HQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.540704801 (376h30m15.424118441s) [noise=true , address=45.91.116.10:4082 , piece_id=S7R3NZTPYD3SX66R3G62KRN6PJMAABDM2WLPUZ5ZICQWLYRJ3GTA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.557783745 (376h30m15.441197365s) [noise=true , address=45.91.116.13:4154 , piece_id=E7PRQBKNZ7VOL32UIH4HSXRZTBRJMUYYZ4DMQKTCS4OPNCBJ27IQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.558361735 (376h30m15.441775352s) [noise=true , address=45.91.116.5:4298 , piece_id=OA2LPGGPRITPZO4VT5REC42ONAEWXDYDMJAQ554HZXI2DJWEPAQQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.565538086 (376h30m15.448951702s) [noise=true , address=77.90.31.10:4518 , piece_id=FC23ZY2HNFKEMN5G3TLNYSJ6QTS57ELGGI5MSOS4OFKXWQJZH5OA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.571767616 (376h30m15.455181304s) [noise=true , address=45.91.116.6:4975 , piece_id=N3LDJ4G6XMWGUY2TEPDY4P5VJWOUMFOJEG3GMIQQONUTSM4UC6HA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.573630878 (376h30m15.457044504s) [noise=true , address=45.91.116.8:4045 , piece_id=CSZWJ3AJVDQHQZVPL3GPAI242XDUNIE52KWGT3KICAL3UA6MJBEA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.580306535 (376h30m15.463720154s) [noise=true , address=77.90.31.16:4733 , piece_id=XV73XHPUUVNC5LMBATS446ILA447SB7QIHBVWL2VBMIQJP5OHB4Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:15.586995475 (376h30m15.470409092s) [noise=true , address=77.90.31.19:4841 , piece_id=DNRBFG2SFYG3DYISXPXE4TJZ7PLKUWUJL2Z2BKR24PVCYWWJDY5Q, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:37.968158647 (376h30m37.851572274s) [noise=true , address=104.36.82.2:4730 , piece_id=H4ER6OZ6FG3H7LWCNTJM2Q5D2FLULXCVWHSAM6KABCAZ2BYEBQCQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: <nil>
Jul 9 01:41:44.413211932 (376h30m44.296625561s) [noise=true , address=104.36.82.7:4911 , piece_id=S7NKW4XURD4AXLWR4MCU34ZMON3QNQPN7AT2LI573QLLLJ6TUQBQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:41:44.413322687 (376h30m44.296736315s) [noise=true , address=104.36.82.3:4767 , piece_id=34FROCZZSLC7RK4XGN7EZX2N2YDJJWEEYKJOP674GR2Z4JWBTVUQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:41:44.413452023 (376h30m44.296865635s) [noise=true , address=104.36.82.4:4801 , piece_id=ETME2T5RJAAVR4WIM5Y3TTARWURQZXZDGP4LXHEQPSUD5GZPAEKA, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54328->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:41:44.413471848 (376h30m44.296885488s) [noise=true , address=104.36.82.8:4969 , piece_id=E3NOPBM4RLETB4A3ZRHN7ISCCFTM37BHFL7XK7QUJGMG6DIA65PQ, seg_pos={0 0}, upload=301082]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50846->104.36.82.8:4969: use of closed network connection
Jul 9 01:41:44.413539946 (376h30m44.29695356s) [seg_pos={0 0}, upload=301082]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:41:44.577747284 (376h30m44.461160908s) [upload=301082]: Done uploading segments...
Jul 9 01:41:44.577784030 (376h30m44.461197666s) [upload=301082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.742234588s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [......X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X] duration:
1m0.00053786s
FAIL (2025-07-09 02:01:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000550075s
core
uplink
us-select-1
us1-placement-20
|