Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
17.733004537s
FAIL (2025-07-09 02:16:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.733037896s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..X..X.............X......X.X.....X.........X...............] duration:
2.901988376s
FAIL (2025-07-09 02:16:44 UTC)
orbiter warning: context deadline exceeded
Duration: 2.902012625s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X.........................X........X........] duration:
26.856617482s
FAIL (2025-07-09 02:16:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 26.856649257s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
45 out of 60 is failed (threshold=3) [X..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXX] duration:
1m0.000631334s
FAIL (2025-07-09 02:16:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000644865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [X..X..X.............X......X.X.....X.........X..............] duration:
9.776271124s
FAIL (2025-07-09 02:16:16 UTC)
orbiter warning: context deadline exceeded
Duration: 9.776293421s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X..X..X.............X......X.X.....X.........X.............] duration:
6.67872232s
FAIL (2025-07-09 02:15:46 UTC)
orbiter warning: context deadline exceeded
Duration: 6.679112208s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
14.53370532s
FAIL (2025-07-09 02:15:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.533736289s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X..X..X.............X......X.X.....X.........X............] duration:
2.840873219s
FAIL (2025-07-09 02:15:19 UTC)
orbiter warning: context deadline exceeded
Duration: 2.84090987s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
45 out of 60 is failed (threshold=3) [XX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXX] duration:
1m0.000743866s
FAIL (2025-07-09 02:15:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000764171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
54.495769385s
FAIL (2025-07-09 02:15:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 54.495796759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X.........................X........X.......] duration:
2.600498479s
FAIL (2025-07-09 02:15:13 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.600554425s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X..X..X.............X......X.X.....X.........X...........] duration:
18.755663217s
FAIL (2025-07-09 02:15:00 UTC)
orbiter warning: context deadline exceeded
Duration: 18.755686654s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X.........................X........X......] duration:
11.038019214s
FAIL (2025-07-09 02:14:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 11.038056988s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....X..X..X.............X......X.X.....X.........X..........] duration:
12.745929638s
FAIL (2025-07-09 02:14:17 UTC)
orbiter warning: context deadline exceeded
Duration: 12.745966025s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
45 out of 60 is failed (threshold=3) [XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XX] duration:
1m0.034541998s
FAIL (2025-07-09 02:14:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.034562267s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X..X..X.............X......X.X.....X.........X.........] duration:
16.749284185s
FAIL (2025-07-09 02:13:53 UTC)
orbiter warning: context deadline exceeded
Duration: 16.749304906s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
26.599138119s
FAIL (2025-07-09 02:13:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 26.599162665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......X..X..X.............X......X.X.....X.........X........] duration:
22.533799478s
FAIL (2025-07-09 02:13:36 UTC)
orbiter warning: context deadline exceeded
Duration: 22.533825987s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X.........................X........X.....] duration:
2.367975977s
FAIL (2025-07-09 02:13:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.368012346s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
48.872039948s
FAIL (2025-07-09 02:13:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 48.8720809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
44 out of 60 is failed (threshold=3) [.XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.X] duration:
1m0.001224171s
FAIL (2025-07-09 02:13:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001257054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X..X..X.............X......X.X.....X.........X.......] duration:
11.919696286s
FAIL (2025-07-09 02:12:41 UTC)
orbiter warning: context deadline exceeded
Duration: 11.919766515s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X.........................X........X....] duration:
2.571324174s
FAIL (2025-07-09 02:12:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.571375452s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
29.320959238s
FAIL (2025-07-09 02:12:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 29.320987895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [..XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.] duration:
26.149008937s
FAIL (2025-07-09 02:12:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.149023838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X.........................X........X...] duration:
2.315327197s
FAIL (2025-07-09 02:11:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.315382861s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXXX] duration:
1m0.000544128s
FAIL (2025-07-09 02:11:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000560036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X..X..X.............X......X.X.....X.........X......] duration:
3.788949888s
FAIL (2025-07-09 02:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 3.788972844s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
13.180258461s
FAIL (2025-07-09 02:11:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.180349816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X.........................X........X..] duration:
2.331323088s
FAIL (2025-07-09 02:10:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.33135254s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
17.001192021s
FAIL (2025-07-09 02:10:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.001239219s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X..X..X.............X......X.X.....X.........X.....] duration:
8.708703999s
FAIL (2025-07-09 02:10:35 UTC)
orbiter warning: context deadline exceeded
Duration: 8.708752319s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXXX] duration:
1m0.000649012s
FAIL (2025-07-09 02:10:30 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000678253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.........................X........X.] duration:
2.322661174s
FAIL (2025-07-09 02:10:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.322742576s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
34.950431337s
FAIL (2025-07-09 02:10:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 34.950474073s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X.........................X........X] duration:
30.106787447s
FAIL (2025-07-09 02:09:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
ssful: 13 optimal: 13 errs: <nil>
Jul 9 02:07:52.076838372 (376h56m51.960251999s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:07:52.076843439 (376h56m51.960257057s) [upload=301441]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:07:52.226972003 (376h56m52.110385642s) [upload=301441]: Done uploading segments...
Jul 9 02:07:52.226997102 (376h56m52.110410733s) [upload=301441]: Upload finished. err: <nil>
Jul 9 02:09:02.139475087 (376h58m2.022888709s) [upload=301458]: Starting upload
Jul 9 02:09:02.139561529 (376h58m2.022975164s) [upload=301458]: Done starting upload
Jul 9 02:09:02.139695306 (376h58m2.023108937s) [upload=301458]: Uploading segments...
Jul 9 02:09:02.140081383 (376h58m2.023495022s) [upload=301458]: Got next segment. Inline: false
Jul 9 02:09:02.140164990 (376h58m2.023578628s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:09:02.140173197 (376h58m2.023586835s) [upload=301458]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:09:02.255230038 (376h58m2.138643679s) [seg_pos={0 0}, upload=301458]: Begin upload segment...
Jul 9 02:09:02.255301800 (376h58m2.13871544s) [seg_pos={0 0}, upload=301458]: Done begin upload segment.
Jul 9 02:09:02.255340167 (376h58m2.13875381s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255481602 (376h58m2.138895233s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.255958363 (376h58m2.139372016s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.256544287 (376h58m2.139957929s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257173310 (376h58m2.140586952s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257667116 (376h58m2.141080757s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.257822693 (376h58m2.141236319s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258205457 (376h58m2.1416191s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258514538 (376h58m2.141928173s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.258648775 (376h58m2.142062417s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259071407 (376h58m2.142485047s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259130551 (376h58m2.142544196s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259508885 (376h58m2.142922527s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.259702194 (376h58m2.143115835s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260249583 (376h58m2.143663222s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.260795687 (376h58m2.144209326s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Uploading piece...
Jul 9 02:09:02.952468521 (376h58m2.83588214s) [upload=301458]: Next returned nil segment
Jul 9 02:09:02.952669666 (376h58m2.836083306s) [upload=301458]: Waiting for error group managing segments...
Jul 9 02:09:03.136423252 (376h58m3.019836884s) [noise=true , address=45.91.116.12:4227 , piece_id=OJZ6BV5EHDWH5NZX3XNTW6IS4UNF7UBRQZ6D3XEEWCVI3XYQJ3YQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.142041871 (376h58m3.02545558s) [noise=true , address=77.90.31.10:4518 , piece_id=HN7P6YUM4DFB6NNCGX6CA5UIB66OF2VRQ3SJ2TMNSHH4NJFQAGIA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.149591556 (376h58m3.033005185s) [noise=true , address=45.91.116.8:4048 , piece_id=IH2FVMSO2PZ4BWKPWOMF2Z7VP3ROWA2RBSV2HM4AV5UMX24U6WHA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.169174234 (376h58m3.052587868s) [noise=true , address=104.36.82.16:4294 , piece_id=EYJRS7NTXNJXNJI53WFED2PF7FMQT6Z76NJRI4OJVTRIXSVTIVPA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.172788671 (376h58m3.056202313s) [noise=true , address=77.90.31.14:4662 , piece_id=H3XZ45HAOSF7QMYCDEK4RKT3IT226WHQB74VZ43TJ65YFUC6C34A, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.184564846 (376h58m3.067978478s) [noise=true , address=45.91.116.13:4155 , piece_id=ABL5ROBYW324I74DHPJO4BPNAONAKDOZTZUVIR7QH5Q2JG3YCFCA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.187229128 (376h58m3.070642744s) [noise=true , address=45.91.116.10:4081 , piece_id=RFZ6PUW3S3WNNFJQJ6UJT7EX7JR7WJRXPMM4M4TADCWPM2GYGCUA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.190934042 (376h58m3.074347665s) [noise=true , address=45.91.116.7:4009 , piece_id=SMAN4ZC6YKJJTDS62XC4ARELLQCIV57VUVEAG7NBBPGLD3USDK2Q, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.197043423 (376h58m3.080457047s) [noise=true , address=77.90.31.15:4697 , piece_id=DO4MCU5WNRSCH4EECLO2JWFAMDABLLKKCEFMUOUFS46YWAHC67TQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.247922965 (376h58m3.131336598s) [noise=true , address=45.91.116.11:4118 , piece_id=6P67ZK4P3M5RHLN6MTEDHFO54RZFA3R6PHFLIVXJUMGM4GKIVMCQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:03.269165573 (376h58m3.152579213s) [noise=true , address=45.91.116.4:4937 , piece_id=I2XMX5S5L6A7NRFH7TYHY2ODTY4BJ4VJ6OSDEAAXWDKPFCSKUYNA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:22.448992117 (376h58m22.332405739s) [noise=true , address=104.36.82.12:4149 , piece_id=KQK47MOBFWOAQ27ZQWT6JNGMPBNRFG7UCV36NHFZODW6WG2M5HKQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: <nil>
Jul 9 02:09:31.855802743 (376h58m31.73921637s) [noise=true , address=104.36.82.8:4969 , piece_id=3KXHBCMLW4X4DSLTD7HTXFZCBFGQDLGQ5KFIZC5O7YGLCBI3PUOQ, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:31.855947029 (376h58m31.73936066s) [noise=true , address=104.36.82.2:4731 , piece_id=AJPPRPTHLXA5RRUTI64TCRCMRFYQY5DDOVKUAGR7QTMC47U6UBAA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59346->104.36.82.2:4731: use of closed network connection
Jul 9 02:09:31.856022986 (376h58m31.739436603s) [noise=true , address=104.36.82.12:4150 , piece_id=MCKUKIAUCRVGL7GUP4SHNKCXVMUNS2PCY4N7GBLO3TZTEUNYE4PA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40298->104.36.82.12:4150: use of closed network connection
Jul 9 02:09:31.856072229 (376h58m31.73948585s) [noise=true , address=104.36.82.6:4874 , piece_id=WCPPHMWGZKITRFFUSXC6XU7WMR27DUUSCMMG2YMRDHE5JSPU63KA, seg_pos={0 0}, upload=301458]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54848->104.36.82.6:4874: use of closed network connection
Jul 9 02:09:31.856095478 (376h58m31.739509104s) [seg_pos={0 0}, upload=301458]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:09:31.960796844 (376h58m31.844210482s) [upload=301458]: Done uploading segments...
Jul 9 02:09:31.960828769 (376h58m31.844242401s) [upload=301458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.106819442s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [.X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXXX] duration:
1m0.00131243s
FAIL (2025-07-09 02:09:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00132433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
33.257368338s
FAIL (2025-07-09 02:09:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 33.257398214s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X..X..X.............X......X.X.....X.........X....] duration:
8.570230245s
FAIL (2025-07-09 02:09:23 UTC)
orbiter warning: context deadline exceeded
Duration: 8.570267107s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........X..X..X.............X......X.X.....X.........X...] duration:
11.353360664s
FAIL (2025-07-09 02:08:32 UTC)
orbiter warning: context deadline exceeded
Duration: 11.353391801s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXXX] duration:
1m0.00055743s
FAIL (2025-07-09 02:08:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000581934s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
35.487323498s
FAIL (2025-07-09 02:08:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 35.4873681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X.........................X........] duration:
2.360538485s
FAIL (2025-07-09 02:07:53 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.360573196s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............X..X..X.............X......X.X.....X.........X..] duration:
2.962340435s
FAIL (2025-07-09 02:07:47 UTC)
orbiter warning: context deadline exceeded
Duration: 2.962379056s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
46.961947337s
FAIL (2025-07-09 02:07:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 46.961974973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXXX] duration:
1m0.001539405s
FAIL (2025-07-09 02:07:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001556419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
45.006731792s
FAIL (2025-07-09 02:06:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
2682347s) [upload=301406]: Got next segment. Inline: false
Jul 9 02:05:12.859345976 (376h54m12.742759591s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:05:12.859352456 (376h54m12.742766083s) [upload=301406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:05:13.062798469 (376h54m12.946212104s) [seg_pos={0 0}, upload=301406]: Begin upload segment...
Jul 9 02:05:13.062848528 (376h54m12.946262144s) [seg_pos={0 0}, upload=301406]: Done begin upload segment.
Jul 9 02:05:13.062880972 (376h54m12.946294606s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063409757 (376h54m12.946823394s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.063733655 (376h54m12.947147282s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064043399 (376h54m12.947457028s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064331313 (376h54m12.947744943s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064630465 (376h54m12.948044092s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.064925260 (376h54m12.948338888s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065262028 (376h54m12.948675657s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065572653 (376h54m12.948986279s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.065861596 (376h54m12.949275229s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066228530 (376h54m12.949642159s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066517718 (376h54m12.949931347s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.066791918 (376h54m12.950205547s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067078518 (376h54m12.950492146s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067439147 (376h54m12.950852783s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.067804191 (376h54m12.951217824s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068085995 (376h54m12.951499623s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.068370813 (376h54m12.951784439s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Uploading piece...
Jul 9 02:05:13.643200688 (376h54m13.526614324s) [upload=301406]: Next returned nil segment
Jul 9 02:05:13.643221951 (376h54m13.526635588s) [upload=301406]: Waiting for error group managing segments...
Jul 9 02:05:13.779005319 (376h54m13.66241895s) [noise=true , address=77.90.31.18:4809 , piece_id=PT6X5PAKXVROVMMVW2AO6GSVP3H3OUL7YN2NQXXIG2MP535BULSQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.779886555 (376h54m13.663300171s) [noise=true , address=77.90.31.19:4871 , piece_id=GIVGKFEFV3RSROBRP6QI2SHXJSLXWLTDDJAKA7B35ONZSB4S5UYQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.785135500 (376h54m13.668549118s) [noise=true , address=77.90.31.11:4574 , piece_id=QMSCG2ZLGMKQJ5CJHII25VWSJDEZ4SPKNYIZYB4U7FZ4EL7S5TBA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.789148935 (376h54m13.672562553s) [noise=true , address=77.90.31.17:4779 , piece_id=XARK5HDA6P4B2RTLFTXM5D4IUGXL3LCYAOSSZLFEAC6KCRBTMVRQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.790782971 (376h54m13.674196586s) [noise=true , address=77.90.31.11:4568 , piece_id=NS3LFY5FX7ZXXZMWNSZSFTCI6FHGISTIB5EYTHREUDDDBORUJSMA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.800708632 (376h54m13.684122259s) [noise=true , address=77.90.31.15:4709 , piece_id=E3W5GPTJJ3W6WLGRUEOGVOW5LAT4YPEU5TH5SY3NV46NRLC6642A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.812912108 (376h54m13.696325743s) [noise=true , address=77.90.31.16:4748 , piece_id=QBIUFRPQD67IWHPJ24NEGJ7O7HQSHTOIKN6DNJIALBIU57CYI2LA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.836234212 (376h54m13.719647835s) [noise=true , address=77.90.31.21:4895 , piece_id=WJDK5L5XHATHIZYMYEZUNWY3MEVMU22BTETYYXN2PLHMQKUTUPAA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:13.876065593 (376h54m13.759479215s) [noise=true , address=45.91.116.12:4240 , piece_id=KIQJYYYGNG747RZDHSS7VFCJJ2L6SRXINO244NQJZ26QLAMYLVCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:14.360410323 (376h54m14.24382394s) [noise=true , address=64.49.13.155:4915 , piece_id=AKOHM4DG5KQ2PTZPJ27ODYE3R6OYDC4DOZR3WAR3NUJNUCJIN5EQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:33.974141397 (376h54m33.857555024s) [noise=true , address=104.36.82.51:4076 , piece_id=R5R5IJUVTIE5COUKIJQT73AOIVZGX5PBJ4WKPISWNQURZWDUEOKQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:34.624728276 (376h54m34.508141903s) [noise=true , address=104.36.82.60:4367 , piece_id=4CEP2WRETWNLQEDO5TFWOQJH43TGJMXBHV6BOP53WOBIGBXQIXCQ, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:05:41.843062744 (376h54m41.72647637s) [noise=true , address=104.36.82.18:4368 , piece_id=ZT7X2LTZPMFKRSI5AR6SF4LT34TFWKJKFI46VQ52XJGOA2STJ64Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: <nil>
Jul 9 02:06:12.586797198 (376h55m12.470210825s) [noise=true , address=104.36.82.38:4696 , piece_id=JNUJIIS7UZRSUHRVJFB7IUZDSVMUPWLCQWXGDTMF7X57GA7DGPIA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: expected piece hash; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47336->104.36.82.38:4696: use of closed network connection; EOF
Jul 9 02:06:12.586926820 (376h55m12.470340447s) [noise=true , address=104.36.82.35:4615 , piece_id=NKI4PUYDB5EPGVQYWRJ3JD4POE7FKOOBX5CCIG77ECNS4WXPVTZA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587005096 (376h55m12.470418726s) [noise=true , address=104.36.82.5:4867 , piece_id=FE6ZQPP6ZKJJZHQ24UILY6CVDLHJQGXFY3XIB4YIK4F4FRXF7V7A, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587035012 (376h55m12.47044863s) [noise=true , address=104.36.82.44:4895 , piece_id=GR32LUIAQF7FS2MIXZLEH57CWGDNXPIEZXHLVZP5TSTHXJ2TXE2Q, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:06:12.587106045 (376h55m12.47051967s) [noise=true , address=104.36.82.19:4424 , piece_id=CH2PVCW6373HOCCOTR7SXIIMJAXQ6YELDDZM776D5O5YU7LZXSYA, seg_pos={0 0}, upload=301406]: Done uploading piece. err: ecclient: upload failed (node:1yH6FFTGsiXqDg52Vrm5jG7pAkojUTQ8ByVGTzr2PkxqymgVij, address:104.36.82.19:4424): protocol: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39004->104.36.82.19:4424: use of closed network connection
Jul 9 02:06:12.587129465 (376h55m12.470543091s) [seg_pos={0 0}, upload=301406]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:06:12.690263902 (376h55m12.573677519s) [upload=301406]: Done uploading segments...
Jul 9 02:06:12.690294026 (376h55m12.573707653s) [upload=301406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 45.006753226s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.........................X.......] duration:
2.288860526s
FAIL (2025-07-09 02:06:51 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:01:39.266404946 (376h50m39.149818572s) [upload=301357]: Done uploading segments...
Jul 9 02:01:39.266424104 (376h50m39.149837719s) [upload=301357]: Upload finished. err: <nil>
Jul 9 02:02:14.513569043 (376h51m14.396982669s) [upload=301363]: Starting upload
Jul 9 02:02:14.513649825 (376h51m14.397063452s) [upload=301363]: Done starting upload
Jul 9 02:02:14.513744112 (376h51m14.397157731s) [upload=301363]: Uploading segments...
Jul 9 02:02:14.514220377 (376h51m14.397633994s) [upload=301363]: Got next segment. Inline: false
Jul 9 02:02:14.514295950 (376h51m14.397709577s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:14.514302738 (376h51m14.397716371s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:14.622203103 (376h51m14.505616739s) [seg_pos={0 0}, upload=301363]: Begin upload segment...
Jul 9 02:02:14.622280622 (376h51m14.505694241s) [seg_pos={0 0}, upload=301363]: Done begin upload segment.
Jul 9 02:02:14.622319757 (376h51m14.505733389s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622329827 (376h51m14.505743454s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622738253 (376h51m14.506151885s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.622817284 (376h51m14.50623091s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623086434 (376h51m14.506500074s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623441663 (376h51m14.506855295s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623734808 (376h51m14.507148436s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.623749205 (376h51m14.507162834s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624071326 (376h51m14.507484955s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624387434 (376h51m14.507801063s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624496623 (376h51m14.507910256s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.624886286 (376h51m14.508299913s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625324367 (376h51m14.508737998s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625646064 (376h51m14.509059693s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.625890551 (376h51m14.509304186s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:14.626210199 (376h51m14.509623821s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Uploading piece...
Jul 9 02:02:15.245220896 (376h51m15.128634527s) [upload=301363]: Next returned nil segment
Jul 9 02:02:15.245241307 (376h51m15.128654944s) [upload=301363]: Waiting for error group managing segments...
Jul 9 02:02:15.374427168 (376h51m15.257840799s) [noise=true , address=45.91.116.11:4119 , piece_id=K6EUJX53ROIJUCGK43EVEBOOY6OJWSZQZTFX6Q7Q4HYQEV466Y7A, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.374492973 (376h51m15.257906602s) [noise=true , address=45.91.116.10:4082 , piece_id=5HO66QCOMGXUZIJ76O2W4W35QPRZSCPTDYJOX6SFHVYL7XYNRP2Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.375608091 (376h51m15.259021703s) [noise=true , address=45.91.116.5:4297 , piece_id=VT4Y5IQSB7AAKE3MT2MOPBK67PU6N36VTRZPS3NX2C5VZKTKA62Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.379430722 (376h51m15.262844359s) [noise=true , address=77.90.31.21:4877 , piece_id=Z4WVV546BVELYVLKMXUZR2CZCMV3ZI47TXVH3JRMF4WXH3GLHQZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.392604028 (376h51m15.27601766s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBV5VHNIE6PKQRNLUYD75DPU22KDL4SGOZ53PGVFRLPS3CTV6SBQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.393229352 (376h51m15.276642972s) [noise=true , address=45.91.116.8:4048 , piece_id=EVDIUFWCYFUH75JRX66RX7DHCFJ5VG3TH4WUJ3HNJB7OUXY3BFQA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394044514 (376h51m15.277458135s) [noise=true , address=77.90.31.15:4697 , piece_id=52YZW43FS6QGYF4JJOGRIEQEEJOXY4QLDJD3NS67PWZPGKN4577Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.394241736 (376h51m15.277655363s) [noise=true , address=77.90.31.10:4518 , piece_id=KH4OCR6QJGVPRZSUMJKMZNOOBBTYGC2TQOVL3AKWOAJ3PBSTSXZQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.400245306 (376h51m15.283658936s) [noise=true , address=104.36.82.9:4006 , piece_id=NZ334TA2TJVYXSDVRLMWDKQR6RCK6JLA7JRGOJXX3ZT7MSVNU5JA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.403998454 (376h51m15.287412066s) [noise=true , address=45.91.116.14:4191 , piece_id=AVBXQAOQXNTSQJSUWC6ZUXXB7ONWIFI4562LEJP6XEW5BXI2OWMA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.406407429 (376h51m15.289821049s) [noise=true , address=77.90.31.17:4771 , piece_id=GGDKIGFEMXDLUK7ONWBFQKJ2WLABTPVSF2RTUWR4GXZLE5P5XYCA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:15.424684329 (376h51m15.308097953s) [noise=true , address=104.36.82.16:4293 , piece_id=RDRCPF5LPBGDEHJWXBLH5PMADPUM33YBOH3HLXTZDIMJQZDL6SSQ, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735411118 (376h51m43.618824737s) [noise=true , address=104.36.82.11:4115 , piece_id=MEKZBUPGQVPRKONEFZJ6LULGO2WM3IYUW5YWH32MBXIPL3N7E3OA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: <nil>
Jul 9 02:02:43.735437104 (376h51m43.618850726s) [seg_pos={0 0}, upload=301363]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:02:43.735876190 (376h51m43.619289819s) [noise=true , address=104.36.82.8:4969 , piece_id=APMLEKI6MJCOWQAQWQARWVL7QUSWM2P6W53BFCQUAZGJDIHRKLZA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52972->104.36.82.8:4969: use of closed network connection
Jul 9 02:02:43.735989611 (376h51m43.61940324s) [noise=true , address=104.36.82.4:4801 , piece_id=JDFXA2Z4N27OG63LGSD4GFXHZVSKB24JZOFSADRV67OX5HWV2PHA, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34328->104.36.82.4:4801: use of closed network connection
Jul 9 02:02:43.736065745 (376h51m43.619479365s) [noise=true , address=104.36.82.6:4873 , piece_id=QVKZYGKP5LUOLNLQX3ZDV5CLMTZAZFZ3VPNVGAOIK5TYRZA7ZV7Q, seg_pos={0 0}, upload=301363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34658->104.36.82.6:4873: use of closed network connection
Jul 9 02:02:43.736091057 (376h51m43.619504684s) [seg_pos={0 0}, upload=301363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:02:43.736177731 (376h51m43.619591357s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:02:43.736182591 (376h51m43.61959622s) [upload=301363]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:02:43.878216721 (376h51m43.761640978s) [upload=301363]: Done uploading segments...
Jul 9 02:02:43.878249238 (376h51m43.761662854s) [upload=301363]: Upload finished. err: <nil>
Duration: 2.28889956s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............X..X..X.............X......X.X.....X.........X.] duration:
8.97175432s
FAIL (2025-07-09 02:06:41 UTC)
orbiter warning: context deadline exceeded
Duration: 8.972365727s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [.X..X...XXX..XXXX.XXXXX..X.XXX.XXXXX.XX.XXXX..XXX.XXX.X.XXXX] duration:
1m0.002144695s
FAIL (2025-07-09 02:06:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.002167009s
core
uplink
us-select-1
us1-placement-20
|