Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 186327 reports (Page 3564 of 3727)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.781094989s FAIL (2025-07-09 21:33:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781119077s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 28.542615802s FAIL (2025-07-09 21:32:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.543001804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................................X.X.....] duration: 18.102145214s FAIL (2025-07-09 21:32:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.102178031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 41.010981669s FAIL (2025-07-09 21:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.011006295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................................X.X....] duration: 35.604003524s FAIL (2025-07-09 21:32:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.604029234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.689798885s FAIL (2025-07-09 21:32:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.689859522s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 36.244126006s FAIL (2025-07-09 21:31:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.244173459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................................X.X...] duration: 18.000351457s FAIL (2025-07-09 21:31:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.000384658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.976838945s FAIL (2025-07-09 21:31:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.976868145s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 35.393166s FAIL (2025-07-09 21:30:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.393176466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.118796268s FAIL (2025-07-09 21:30:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.118818434s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................................X.X..] duration: 22.831821692s FAIL (2025-07-09 21:30:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.831914497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 25.299902736s FAIL (2025-07-09 21:30:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.299920668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................................X.X.] duration: 39.182570093s FAIL (2025-07-09 21:30:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 39.182600271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 39.690950634s FAIL (2025-07-09 21:29:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.690978661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.288332982s FAIL (2025-07-09 21:29:53 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.288366166s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.909048814s FAIL (2025-07-09 21:29:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.909070745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........................................X.X] duration: 1m0.112494849s FAIL (2025-07-09 21:29:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1:28:16.399511268 (396h17m16.282924895s) [upload=316906]: Upload finished. err: <nil> Jul 9 21:28:23.538291019 (396h17m23.421704645s) [upload=316913]: Starting upload Jul 9 21:28:23.538388210 (396h17m23.421801953s) [upload=316913]: Done starting upload Jul 9 21:28:23.538561128 (396h17m23.421974756s) [upload=316913]: Uploading segments... Jul 9 21:28:23.538713671 (396h17m23.422127288s) [upload=316913]: Got next segment. Inline: false Jul 9 21:28:23.538778392 (396h17m23.422192021s) [upload=316913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:28:23.538784967 (396h17m23.422198593s) [upload=316913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:28:23.740137051 (396h17m23.623550668s) [seg_pos={0 0}, upload=316913]: Begin upload segment... Jul 9 21:28:23.740213065 (396h17m23.62362668s) [seg_pos={0 0}, upload=316913]: Done begin upload segment. Jul 9 21:28:23.740265203 (396h17m23.623678819s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740716950 (396h17m23.624130585s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.740939545 (396h17m23.624353181s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741065709 (396h17m23.624479334s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741566059 (396h17m23.624979695s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.741833163 (396h17m23.6252468s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742368882 (396h17m23.625782508s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.742552561 (396h17m23.62596619s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743110701 (396h17m23.626524332s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743207937 (396h17m23.626621568s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743713286 (396h17m23.627126914s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.743828588 (396h17m23.627242217s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744448921 (396h17m23.627862555s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744499733 (396h17m23.627913368s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.744929138 (396h17m23.62834277s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746158995 (396h17m23.629572631s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746494859 (396h17m23.629908486s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:23.746915419 (396h17m23.630329054s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Uploading piece... Jul 9 21:28:24.327360618 (396h17m24.210774253s) [upload=316913]: Next returned nil segment Jul 9 21:28:24.327379424 (396h17m24.21079306s) [upload=316913]: Waiting for error group managing segments... Jul 9 21:28:24.467627976 (396h17m24.351041594s) [noise=true , address=77.90.31.12:4598 , piece_id=SDLPAQNLSOG44OQSRY42OY4ZJA5VLDPFBBYVXPGLLDILUCOHMXFQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.467671774 (396h17m24.351085395s) [noise=true , address=77.90.31.19:4870 , piece_id=ZTCR42KYT3CABHFSIKFBFZAZSESWZOKHPVQUEQCZ6QZ3KIHW2XSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.472808225 (396h17m24.35622184s) [noise=true , address=77.90.31.15:4720 , piece_id=W5Z7W3ULVZV3DSC42LFODIRDLCPGEQULVSH47BAPV7NYZBXBZDJA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.473036104 (396h17m24.356449724s) [noise=true , address=77.90.31.18:4820 , piece_id=SXNEYWHIITJB2JIZLW2MI7RJ4MAKNGXMAXTSX2MVCCIOL7SYVNSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.478247346 (396h17m24.36166096s) [noise=true , address=77.90.31.11:4557 , piece_id=2WCPQDOP4S73VERMMVRNOPXAJBPHCNNRTUHIU2HUYVWSCBRT4BIQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.483244247 (396h17m24.366657865s) [noise=true , address=77.90.31.10:4535 , piece_id=RDHPP3YDAVH7HE7UXKLMTP34RMVEQHTW4GYFTZ25AZSSFZ6IIRFA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.492583188 (396h17m24.375996805s) [noise=true , address=77.90.31.17:4778 , piece_id=Y7P2XBVWPZTBMCDJTDGZYN7SUJ6GPMCX37YCLZP7BWJCKR2LOUZA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:24.513871599 (396h17m24.39728523s) [noise=true , address=77.90.31.19:4876 , piece_id=BMWGE5XZBUK5BQGKF6NNYGVJF2OAO4FVH3Q2GXD7GNNTDEF6DN4A, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:32.835757927 (396h17m32.719171539s) [noise=true , address=104.36.82.8:4978 , piece_id=UAKILTPKGASX673XSPM26SPLS7LRZEO4S3TA2MSWKP44MVJQ4CTA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:28:42.677433957 (396h17m42.560847574s) [noise=true , address=104.36.82.17:4349 , piece_id=FX5D5HQH4ATG6CX46IVSMJYCFSHTUSADRGX76FNHYRSTBMRMVMAQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.197024743 (396h18m3.080438356s) [noise=true , address=45.91.95.2:4028 , piece_id=KWCOYY3RTMA7VCV6TUA62BPRKDXLFZZ572WLJI3QTHMHWKXFQ5PA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:03.888475907 (396h18m3.771889522s) [noise=true , address=104.36.82.7:4937 , piece_id=26KK45MO7TXMWVNDZT257HWRCFVDO7MBVWJDQW2JIANRRXRZRHSQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:04.499512425 (396h18m4.382926041s) [noise=true , address=104.36.82.36:4652 , piece_id=CAWMFUL7R7TPCTU7FM7ABHBGAFOLEET46XCS5HUCJGTTYXFN7GYQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: <nil> Jul 9 21:29:23.267688433 (396h18m23.151102047s) [noise=true , address=45.91.95.5:4003 , piece_id=2H7ECDAIXIUQQ2BKKOGAMLRINTDADBBZP6SV5OQXQN3MGKMMOEOQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7, address:45.91.95.5:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267793830 (396h18m23.151207444s) [noise=true , address=104.36.82.18:4376 , piece_id=XM3E2RHBEJNUGPKYOVU3PDCQQEI7SEL7PXD5KIWKFT2TJRBHEM4Q, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267816131 (396h18m23.151229749s) [noise=true , address=64.49.13.169:4706 , piece_id=S2JBFPLUW75CIMTWAMS23YJLVQK4P5VIQXVWX75OR3474QQOHDQA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267867052 (396h18m23.151280673s) [noise=true , address=45.91.95.4:4010 , piece_id=XTXT5OCXCKGZH45OHVNULGMXP63ADTUTA3BBC73VKKZTGCBEP6LA, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:129tyPaFUJBVtavq2kR9ZmPit8FpMMUNcQDaPxtSjbyucwze52G, address:45.91.95.4:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.267977183 (396h18m23.151390813s) [noise=true , address=45.91.95.3:4015 , piece_id=PHE4J7JEA66XOFDTK5CN4SKJPNZUKFHRAVGVOBKQZPMQYNERITHQ, seg_pos={0 0}, upload=316913]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:29:23.268019233 (396h18m23.151432869s) [seg_pos={0 0}, upload=316913]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:29:23.377805174 (396h18m23.261218801s) [upload=316913]: Done uploading segments... Jul 9 21:29:23.377832620 (396h18m23.261246247s) [upload=316913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.112514974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 41.501584622s FAIL (2025-07-09 21:29:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.501603394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.339531259s FAIL (2025-07-09 21:28:46 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.339581355s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.811440477s FAIL (2025-07-09 21:28:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811467434s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 24.569298951s FAIL (2025-07-09 21:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.569498091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................................X.] duration: 35.445394878s FAIL (2025-07-09 21:28:23 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=316896]: Starting upload Jul 9 21:26:47.791560946 (396h15m47.674974576s) [upload=316896]: Done starting upload Jul 9 21:26:47.791671404 (396h15m47.675085042s) [upload=316896]: Uploading segments... Jul 9 21:26:47.792890744 (396h15m47.676304364s) [upload=316896]: Got next segment. Inline: false Jul 9 21:26:47.793022182 (396h15m47.676435822s) [upload=316896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:26:47.793035263 (396h15m47.676448901s) [upload=316896]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:26:48.065580968 (396h15m47.948994604s) [seg_pos={0 0}, upload=316896]: Begin upload segment... Jul 9 21:26:48.065688663 (396h15m47.949102316s) [noise=true , address=45.91.95.4:4014 , piece_id=5HSSVRXZ3SOMIGAYACSNZ4MTOEUGRFETFAM3CYVVSHMMT3WSUSAA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.065729985 (396h15m47.94914363s) [noise=true , address=104.36.82.34:4546 , piece_id=BZ5JG2JRRXWN57D7C62ESEJDRXWO32POR3YMNFUB3RUM6SFWTDXA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.066722896 (396h15m47.950136535s) [noise=true , address=104.36.82.38:4716 , piece_id=6OS7OMICBLJGMRYM3FWIHUXR4B2JILZWBAYJTKENUS672JGLFXVQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.066940559 (396h15m47.950354199s) [noise=true , address=45.91.95.5:4036 , piece_id=RRQFBJQ6PAIOITGDCRSJKZV6N7DU37NA2AJA57ALDPJWTCHRANUA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.067554840 (396h15m47.950968461s) [seg_pos={0 0}, upload=316896]: Done begin upload segment. Jul 9 21:26:48.068065307 (396h15m47.951478942s) [noise=true , address=79.127.219.47:4044 , piece_id=5JZKAKU6BWR2QNBSPJG73PLGFMO3CV4PMTEINHXESVCWYTKZOBNQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.073245404 (396h15m47.956659042s) [noise=true , address=77.90.31.12:4616 , piece_id=P23C2W27PVZ4QRQWC4D5ZHRTJJYTBCHGR7WK2DXAXPJI7ZOI7AOA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.074776060 (396h15m47.9581897s) [noise=true , address=104.36.82.41:4793 , piece_id=CHZX2UPG75IEV3AUALAI76UC7RRSZYOPXZ64H6PJXWMZBTFMSY6Q, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.075478892 (396h15m47.958892528s) [noise=true , address=77.90.31.17:4774 , piece_id=APZ3HOLHVTDM7WH2ZKL7YJPC4MMAFS226VQSSLMKV7S4KK3L65YQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.076303206 (396h15m47.959716843s) [noise=true , address=104.36.82.2:4751 , piece_id=6A75HNC4GQAY2EKJCMT7M667S7UZKY6VVIRISCPPCO2LEYCWV3JQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.077228565 (396h15m47.960642204s) [noise=true , address=77.90.31.18:4822 , piece_id=WT5GXYAWHAM4JUEALDPP3UAVA2YTFBECXSQHCXW2UKMZSVVG42SA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.078610364 (396h15m47.962023999s) [noise=true , address=104.36.82.23:4464 , piece_id=YQSEXDCLPWE36ZOCG3BDHKUTPQDKBGRSMJPI655LOQXEFSM7CCYA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.079112861 (396h15m47.962526516s) [noise=true , address=77.90.31.14:4677 , piece_id=YUUDGHUB2RIAH6MJ2BRR5ANXJSBAVXXMH2ILKRK4XHQN2UIRSIRQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.079591650 (396h15m47.963005284s) [noise=true , address=77.90.31.13:4634 , piece_id=ADFM736VIAEPCYAW3RL5WZOCDY3RWDH72EZNEDEOXDQW4JHCI5GA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.080117772 (396h15m47.963531408s) [noise=true , address=45.91.95.2:4018 , piece_id=L7J7RRKDN4PTNH4FK2K65NNHUPO4O5B6RBJH7OLWPTPC7UBXFCMA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.080842435 (396h15m47.964256071s) [noise=true , address=104.36.82.19:4434 , piece_id=ZVGMSQE73YKOBZK524RFVVDNYUVMMOFAYCSJPAQ2S67RV2CNTKIQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.082015272 (396h15m47.965428908s) [noise=true , address=79.127.219.38:4032 , piece_id=LEL6OU5NSDFMPYQFDWF5HVSKZZ5ZRVTJNCMN45O62P3G4KFROMXA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.082662004 (396h15m47.96607564s) [noise=true , address=77.90.31.21:4888 , piece_id=2RKZ6OD2ITKPQ4RSP5X7CDVAXVZUNLDBA73L3HSRL45PZUU5NQ5A, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.084639150 (396h15m47.968052788s) [noise=true , address=104.36.82.18:4396 , piece_id=QUNZ5GCD3NWNRCGRA3OL5BZPTGZQZDUD3QOIE2LM3WDXQISYWQPA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.661567142 (396h15m48.544980769s) [upload=316896]: Next returned nil segment Jul 9 21:26:48.661586112 (396h15m48.54499975s) [upload=316896]: Waiting for error group managing segments... Jul 9 21:26:48.801758765 (396h15m48.685172384s) [noise=true , address=77.90.31.18:4822 , piece_id=WT5GXYAWHAM4JUEALDPP3UAVA2YTFBECXSQHCXW2UKMZSVVG42SA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.802378338 (396h15m48.685791958s) [noise=true , address=77.90.31.17:4774 , piece_id=APZ3HOLHVTDM7WH2ZKL7YJPC4MMAFS226VQSSLMKV7S4KK3L65YQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.812295645 (396h15m48.695709277s) [noise=true , address=77.90.31.14:4677 , piece_id=YUUDGHUB2RIAH6MJ2BRR5ANXJSBAVXXMH2ILKRK4XHQN2UIRSIRQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.818159636 (396h15m48.701573269s) [noise=true , address=77.90.31.12:4616 , piece_id=P23C2W27PVZ4QRQWC4D5ZHRTJJYTBCHGR7WK2DXAXPJI7ZOI7AOA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.835233386 (396h15m48.718646998s) [noise=true , address=79.127.219.47:4044 , piece_id=5JZKAKU6BWR2QNBSPJG73PLGFMO3CV4PMTEINHXESVCWYTKZOBNQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.840792932 (396h15m48.724206571s) [noise=true , address=77.90.31.13:4634 , piece_id=ADFM736VIAEPCYAW3RL5WZOCDY3RWDH72EZNEDEOXDQW4JHCI5GA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.850959807 (396h15m48.734373467s) [noise=true , address=77.90.31.21:4888 , piece_id=2RKZ6OD2ITKPQ4RSP5X7CDVAXVZUNLDBA73L3HSRL45PZUU5NQ5A, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:50.549675010 (396h15m50.433088622s) [noise=true , address=45.91.95.5:4036 , piece_id=RRQFBJQ6PAIOITGDCRSJKZV6N7DU37NA2AJA57ALDPJWTCHRANUA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: ecclient: upload failed (node:15FVyNdBfSbRz7agQ2EsXWhEM6BEdAutJrtTrfyRiXURHRTqhf, address:45.91.95.5:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:26:52.466220922 (396h15m52.34963455s) [noise=true , address=45.91.95.4:4014 , piece_id=5HSSVRXZ3SOMIGAYACSNZ4MTOEUGRFETFAM3CYVVSHMMT3WSUSAA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: ecclient: upload failed (node:126AYzCEC5z4NMonMoPAV15R2KeYLDfimhiPZrhzhTHJRZnNysz, address:45.91.95.4:4014): protocol: storage node overloaded, request limit: 2000 Jul 9 21:27:01.099463097 (396h16m0.982876735s) [noise=true , address=79.127.219.38:4032 , piece_id=LEL6OU5NSDFMPYQFDWF5HVSKZZ5ZRVTJNCMN45O62P3G4KFROMXA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:19.114069983 (396h16m18.997483615s) [noise=true , address=104.36.82.41:4793 , piece_id=CHZX2UPG75IEV3AUALAI76UC7RRSZYOPXZ64H6PJXWMZBTFMSY6Q, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:19.212830696 (396h16m19.096244317s) [noise=true , address=104.36.82.38:4716 , piece_id=6OS7OMICBLJGMRYM3FWIHUXR4B2JILZWBAYJTKENUS672JGLFXVQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:28.302523576 (396h16m28.185937193s) [noise=true , address=104.36.82.18:4396 , piece_id=QUNZ5GCD3NWNRCGRA3OL5BZPTGZQZDUD3QOIE2LM3WDXQISYWQPA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:38.006858398 (396h16m37.890272011s) [noise=true , address=104.36.82.2:4751 , piece_id=6A75HNC4GQAY2EKJCMT7M667S7UZKY6VVIRISCPPCO2LEYCWV3JQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:43.767562996 (396h16m43.650976622s) [noise=true , address=104.36.82.19:4434 , piece_id=ZVGMSQE73YKOBZK524RFVVDNYUVMMOFAYCSJPAQ2S67RV2CNTKIQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:44.115273432 (396h16m43.99868705s) [noise=true , address=104.36.82.34:4546 , piece_id=BZ5JG2JRRXWN57D7C62ESEJDRXWO32POR3YMNFUB3RUM6SFWTDXA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:44.115293835 (396h16m43.998707453s) [seg_pos={0 0}, upload=316896]: Segment reached optimal threshold of 14 pieces. Jul 9 21:27:44.115449386 (396h16m43.998863006s) [noise=true , address=45.91.95.2:4018 , piece_id=L7J7RRKDN4PTNH4FK2K65NNHUPO4O5B6RBJH7OLWPTPC7UBXFCMA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi): protocol: expected piece hash; context canceled; EOF Jul 9 21:27:44.115624250 (396h16m43.999037885s) [noise=true , address=104.36.82.23:4464 , piece_id=YQSEXDCLPWE36ZOCG3BDHKUTPQDKBGRSMJPI655LOQXEFSM7CCYA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk): protocol: expected piece hash; context canceled; EOF Jul 9 21:27:44.115660319 (396h16m43.999073956s) [seg_pos={0 0}, upload=316896]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:27:44.115767292 (396h16m43.999180927s) [upload=316896]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:27:44.115791958 (396h16m43.999205595s) [upload=316896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:27:44.277264867 (396h16m44.160678497s) [upload=316896]: Done uploading segments... Jul 9 21:27:44.277289599 (396h16m44.160703231s) [upload=316896]: Upload finished. err: <nil> Duration: 35.445417993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.52434274s FAIL (2025-07-09 21:28:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.524389917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 1m0.017979413s FAIL (2025-07-09 21:27:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=316896]: Starting upload Jul 9 21:26:47.791560946 (396h15m47.674974576s) [upload=316896]: Done starting upload Jul 9 21:26:47.791671404 (396h15m47.675085042s) [upload=316896]: Uploading segments... Jul 9 21:26:47.792890744 (396h15m47.676304364s) [upload=316896]: Got next segment. Inline: false Jul 9 21:26:47.793022182 (396h15m47.676435822s) [upload=316896]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:26:47.793035263 (396h15m47.676448901s) [upload=316896]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:26:48.065580968 (396h15m47.948994604s) [seg_pos={0 0}, upload=316896]: Begin upload segment... Jul 9 21:26:48.065688663 (396h15m47.949102316s) [noise=true , address=45.91.95.4:4014 , piece_id=5HSSVRXZ3SOMIGAYACSNZ4MTOEUGRFETFAM3CYVVSHMMT3WSUSAA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.065729985 (396h15m47.94914363s) [noise=true , address=104.36.82.34:4546 , piece_id=BZ5JG2JRRXWN57D7C62ESEJDRXWO32POR3YMNFUB3RUM6SFWTDXA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.066722896 (396h15m47.950136535s) [noise=true , address=104.36.82.38:4716 , piece_id=6OS7OMICBLJGMRYM3FWIHUXR4B2JILZWBAYJTKENUS672JGLFXVQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.066940559 (396h15m47.950354199s) [noise=true , address=45.91.95.5:4036 , piece_id=RRQFBJQ6PAIOITGDCRSJKZV6N7DU37NA2AJA57ALDPJWTCHRANUA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.067554840 (396h15m47.950968461s) [seg_pos={0 0}, upload=316896]: Done begin upload segment. Jul 9 21:26:48.068065307 (396h15m47.951478942s) [noise=true , address=79.127.219.47:4044 , piece_id=5JZKAKU6BWR2QNBSPJG73PLGFMO3CV4PMTEINHXESVCWYTKZOBNQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.073245404 (396h15m47.956659042s) [noise=true , address=77.90.31.12:4616 , piece_id=P23C2W27PVZ4QRQWC4D5ZHRTJJYTBCHGR7WK2DXAXPJI7ZOI7AOA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.074776060 (396h15m47.9581897s) [noise=true , address=104.36.82.41:4793 , piece_id=CHZX2UPG75IEV3AUALAI76UC7RRSZYOPXZ64H6PJXWMZBTFMSY6Q, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.075478892 (396h15m47.958892528s) [noise=true , address=77.90.31.17:4774 , piece_id=APZ3HOLHVTDM7WH2ZKL7YJPC4MMAFS226VQSSLMKV7S4KK3L65YQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.076303206 (396h15m47.959716843s) [noise=true , address=104.36.82.2:4751 , piece_id=6A75HNC4GQAY2EKJCMT7M667S7UZKY6VVIRISCPPCO2LEYCWV3JQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.077228565 (396h15m47.960642204s) [noise=true , address=77.90.31.18:4822 , piece_id=WT5GXYAWHAM4JUEALDPP3UAVA2YTFBECXSQHCXW2UKMZSVVG42SA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.078610364 (396h15m47.962023999s) [noise=true , address=104.36.82.23:4464 , piece_id=YQSEXDCLPWE36ZOCG3BDHKUTPQDKBGRSMJPI655LOQXEFSM7CCYA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.079112861 (396h15m47.962526516s) [noise=true , address=77.90.31.14:4677 , piece_id=YUUDGHUB2RIAH6MJ2BRR5ANXJSBAVXXMH2ILKRK4XHQN2UIRSIRQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.079591650 (396h15m47.963005284s) [noise=true , address=77.90.31.13:4634 , piece_id=ADFM736VIAEPCYAW3RL5WZOCDY3RWDH72EZNEDEOXDQW4JHCI5GA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.080117772 (396h15m47.963531408s) [noise=true , address=45.91.95.2:4018 , piece_id=L7J7RRKDN4PTNH4FK2K65NNHUPO4O5B6RBJH7OLWPTPC7UBXFCMA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.080842435 (396h15m47.964256071s) [noise=true , address=104.36.82.19:4434 , piece_id=ZVGMSQE73YKOBZK524RFVVDNYUVMMOFAYCSJPAQ2S67RV2CNTKIQ, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.082015272 (396h15m47.965428908s) [noise=true , address=79.127.219.38:4032 , piece_id=LEL6OU5NSDFMPYQFDWF5HVSKZZ5ZRVTJNCMN45O62P3G4KFROMXA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.082662004 (396h15m47.96607564s) [noise=true , address=77.90.31.21:4888 , piece_id=2RKZ6OD2ITKPQ4RSP5X7CDVAXVZUNLDBA73L3HSRL45PZUU5NQ5A, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.084639150 (396h15m47.968052788s) [noise=true , address=104.36.82.18:4396 , piece_id=QUNZ5GCD3NWNRCGRA3OL5BZPTGZQZDUD3QOIE2LM3WDXQISYWQPA, seg_pos={0 0}, upload=316896]: Uploading piece... Jul 9 21:26:48.661567142 (396h15m48.544980769s) [upload=316896]: Next returned nil segment Jul 9 21:26:48.661586112 (396h15m48.54499975s) [upload=316896]: Waiting for error group managing segments... Jul 9 21:26:48.801758765 (396h15m48.685172384s) [noise=true , address=77.90.31.18:4822 , piece_id=WT5GXYAWHAM4JUEALDPP3UAVA2YTFBECXSQHCXW2UKMZSVVG42SA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.802378338 (396h15m48.685791958s) [noise=true , address=77.90.31.17:4774 , piece_id=APZ3HOLHVTDM7WH2ZKL7YJPC4MMAFS226VQSSLMKV7S4KK3L65YQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.812295645 (396h15m48.695709277s) [noise=true , address=77.90.31.14:4677 , piece_id=YUUDGHUB2RIAH6MJ2BRR5ANXJSBAVXXMH2ILKRK4XHQN2UIRSIRQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.818159636 (396h15m48.701573269s) [noise=true , address=77.90.31.12:4616 , piece_id=P23C2W27PVZ4QRQWC4D5ZHRTJJYTBCHGR7WK2DXAXPJI7ZOI7AOA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.835233386 (396h15m48.718646998s) [noise=true , address=79.127.219.47:4044 , piece_id=5JZKAKU6BWR2QNBSPJG73PLGFMO3CV4PMTEINHXESVCWYTKZOBNQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.840792932 (396h15m48.724206571s) [noise=true , address=77.90.31.13:4634 , piece_id=ADFM736VIAEPCYAW3RL5WZOCDY3RWDH72EZNEDEOXDQW4JHCI5GA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:48.850959807 (396h15m48.734373467s) [noise=true , address=77.90.31.21:4888 , piece_id=2RKZ6OD2ITKPQ4RSP5X7CDVAXVZUNLDBA73L3HSRL45PZUU5NQ5A, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:26:50.549675010 (396h15m50.433088622s) [noise=true , address=45.91.95.5:4036 , piece_id=RRQFBJQ6PAIOITGDCRSJKZV6N7DU37NA2AJA57ALDPJWTCHRANUA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: ecclient: upload failed (node:15FVyNdBfSbRz7agQ2EsXWhEM6BEdAutJrtTrfyRiXURHRTqhf, address:45.91.95.5:4036): protocol: storage node overloaded, request limit: 2000 Jul 9 21:26:52.466220922 (396h15m52.34963455s) [noise=true , address=45.91.95.4:4014 , piece_id=5HSSVRXZ3SOMIGAYACSNZ4MTOEUGRFETFAM3CYVVSHMMT3WSUSAA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: ecclient: upload failed (node:126AYzCEC5z4NMonMoPAV15R2KeYLDfimhiPZrhzhTHJRZnNysz, address:45.91.95.4:4014): protocol: storage node overloaded, request limit: 2000 Jul 9 21:27:01.099463097 (396h16m0.982876735s) [noise=true , address=79.127.219.38:4032 , piece_id=LEL6OU5NSDFMPYQFDWF5HVSKZZ5ZRVTJNCMN45O62P3G4KFROMXA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:19.114069983 (396h16m18.997483615s) [noise=true , address=104.36.82.41:4793 , piece_id=CHZX2UPG75IEV3AUALAI76UC7RRSZYOPXZ64H6PJXWMZBTFMSY6Q, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:19.212830696 (396h16m19.096244317s) [noise=true , address=104.36.82.38:4716 , piece_id=6OS7OMICBLJGMRYM3FWIHUXR4B2JILZWBAYJTKENUS672JGLFXVQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:28.302523576 (396h16m28.185937193s) [noise=true , address=104.36.82.18:4396 , piece_id=QUNZ5GCD3NWNRCGRA3OL5BZPTGZQZDUD3QOIE2LM3WDXQISYWQPA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:38.006858398 (396h16m37.890272011s) [noise=true , address=104.36.82.2:4751 , piece_id=6A75HNC4GQAY2EKJCMT7M667S7UZKY6VVIRISCPPCO2LEYCWV3JQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:43.767562996 (396h16m43.650976622s) [noise=true , address=104.36.82.19:4434 , piece_id=ZVGMSQE73YKOBZK524RFVVDNYUVMMOFAYCSJPAQ2S67RV2CNTKIQ, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:44.115273432 (396h16m43.99868705s) [noise=true , address=104.36.82.34:4546 , piece_id=BZ5JG2JRRXWN57D7C62ESEJDRXWO32POR3YMNFUB3RUM6SFWTDXA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: <nil> Jul 9 21:27:44.115293835 (396h16m43.998707453s) [seg_pos={0 0}, upload=316896]: Segment reached optimal threshold of 14 pieces. Jul 9 21:27:44.115449386 (396h16m43.998863006s) [noise=true , address=45.91.95.2:4018 , piece_id=L7J7RRKDN4PTNH4FK2K65NNHUPO4O5B6RBJH7OLWPTPC7UBXFCMA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi): protocol: expected piece hash; context canceled; EOF Jul 9 21:27:44.115624250 (396h16m43.999037885s) [noise=true , address=104.36.82.23:4464 , piece_id=YQSEXDCLPWE36ZOCG3BDHKUTPQDKBGRSMJPI655LOQXEFSM7CCYA, seg_pos={0 0}, upload=316896]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk): protocol: expected piece hash; context canceled; EOF Jul 9 21:27:44.115660319 (396h16m43.999073956s) [seg_pos={0 0}, upload=316896]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:27:44.115767292 (396h16m43.999180927s) [upload=316896]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:27:44.115791958 (396h16m43.999205595s) [upload=316896]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:27:44.277264867 (396h16m44.160678497s) [upload=316896]: Done uploading segments... Jul 9 21:27:44.277289599 (396h16m44.160703231s) [upload=316896]: Upload finished. err: <nil> Duration: 1m0.018067894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 52.620872325s FAIL (2025-07-09 21:27:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.620895084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.291931909s FAIL (2025-07-09 21:27:34 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.291960519s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.819681116s FAIL (2025-07-09 21:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819711599s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.784543314s FAIL (2025-07-09 21:26:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.78456895s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 48.411837597s FAIL (2025-07-09 21:26:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.41186628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.329226953s FAIL (2025-07-09 21:26:23 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.329259642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 17.034070867s FAIL (2025-07-09 21:26:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ul 9 20:53:05.181732432 (395h42m5.065146066s) [upload=316437]: Upload finished. err: <nil> Jul 9 20:53:46.239759806 (395h42m46.123173434s) [upload=316447]: Starting upload Jul 9 20:53:46.239879159 (395h42m46.123292791s) [upload=316447]: Done starting upload Jul 9 20:53:46.240039985 (395h42m46.123453618s) [upload=316447]: Uploading segments... Jul 9 20:53:46.240311285 (395h42m46.123724915s) [upload=316447]: Got next segment. Inline: false Jul 9 20:53:46.240364403 (395h42m46.123778042s) [upload=316447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:53:46.240374251 (395h42m46.123787889s) [upload=316447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:53:46.440120932 (395h42m46.323534548s) [seg_pos={0 0}, upload=316447]: Begin upload segment... Jul 9 20:53:46.440179950 (395h42m46.323593568s) [seg_pos={0 0}, upload=316447]: Done begin upload segment. Jul 9 20:53:46.440217375 (395h42m46.323631001s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440650993 (395h42m46.32406462s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440924286 (395h42m46.324337911s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441186006 (395h42m46.324599632s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441444482 (395h42m46.324858108s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441690913 (395h42m46.325104542s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442118579 (395h42m46.325532246s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442302181 (395h42m46.32571581s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442705594 (395h42m46.326119222s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443169151 (395h42m46.326582779s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443516420 (395h42m46.326930048s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443951987 (395h42m46.327365616s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444246383 (395h42m46.32766001s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444526835 (395h42m46.327940466s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444714260 (395h42m46.328127876s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445082976 (395h42m46.328496603s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445455193 (395h42m46.328868819s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.446344122 (395h42m46.329757751s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:47.020800550 (395h42m46.90421418s) [upload=316447]: Next returned nil segment Jul 9 20:53:47.020819787 (395h42m46.904233401s) [upload=316447]: Waiting for error group managing segments... Jul 9 20:53:47.156598268 (395h42m47.04001189s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.159389097 (395h42m47.042802716s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.160887797 (395h42m47.044301415s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.163302365 (395h42m47.046715982s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.167069778 (395h42m47.050483398s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.168030609 (395h42m47.051444228s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.383117245 (395h42m47.266530862s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:48.832058921 (395h42m48.71547254s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:49.721740279 (395h42m49.605153896s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: storage node overloaded, request limit: 2000 Jul 9 20:53:58.022618980 (395h42m57.9060326s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:03.273249375 (395h43m3.156663002s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:09.469438565 (395h43m9.352852186s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:10.167123637 (395h43m10.050537256s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:38.676415445 (395h43m38.559829065s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:45.960480350 (395h43m45.843894004s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960513865 (395h43m45.843927481s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960590132 (395h43m45.844003749s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960601510 (395h43m45.844015126s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960620902 (395h43m45.844034518s) [seg_pos={0 0}, upload=316447]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:54:46.067458842 (395h43m45.950872454s) [upload=316447]: Done uploading segments... Jul 9 20:54:46.067486298 (395h43m45.950899914s) [upload=316447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.034104008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 15.585071549s FAIL (2025-07-09 21:26:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.585123504s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 18.382820411s FAIL (2025-07-09 21:25:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ul 9 20:53:05.181732432 (395h42m5.065146066s) [upload=316437]: Upload finished. err: <nil> Jul 9 20:53:46.239759806 (395h42m46.123173434s) [upload=316447]: Starting upload Jul 9 20:53:46.239879159 (395h42m46.123292791s) [upload=316447]: Done starting upload Jul 9 20:53:46.240039985 (395h42m46.123453618s) [upload=316447]: Uploading segments... Jul 9 20:53:46.240311285 (395h42m46.123724915s) [upload=316447]: Got next segment. Inline: false Jul 9 20:53:46.240364403 (395h42m46.123778042s) [upload=316447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:53:46.240374251 (395h42m46.123787889s) [upload=316447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:53:46.440120932 (395h42m46.323534548s) [seg_pos={0 0}, upload=316447]: Begin upload segment... Jul 9 20:53:46.440179950 (395h42m46.323593568s) [seg_pos={0 0}, upload=316447]: Done begin upload segment. Jul 9 20:53:46.440217375 (395h42m46.323631001s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440650993 (395h42m46.32406462s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440924286 (395h42m46.324337911s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441186006 (395h42m46.324599632s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441444482 (395h42m46.324858108s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441690913 (395h42m46.325104542s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442118579 (395h42m46.325532246s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442302181 (395h42m46.32571581s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442705594 (395h42m46.326119222s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443169151 (395h42m46.326582779s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443516420 (395h42m46.326930048s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443951987 (395h42m46.327365616s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444246383 (395h42m46.32766001s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444526835 (395h42m46.327940466s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444714260 (395h42m46.328127876s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445082976 (395h42m46.328496603s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445455193 (395h42m46.328868819s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.446344122 (395h42m46.329757751s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:47.020800550 (395h42m46.90421418s) [upload=316447]: Next returned nil segment Jul 9 20:53:47.020819787 (395h42m46.904233401s) [upload=316447]: Waiting for error group managing segments... Jul 9 20:53:47.156598268 (395h42m47.04001189s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.159389097 (395h42m47.042802716s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.160887797 (395h42m47.044301415s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.163302365 (395h42m47.046715982s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.167069778 (395h42m47.050483398s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.168030609 (395h42m47.051444228s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.383117245 (395h42m47.266530862s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:48.832058921 (395h42m48.71547254s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:49.721740279 (395h42m49.605153896s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: storage node overloaded, request limit: 2000 Jul 9 20:53:58.022618980 (395h42m57.9060326s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:03.273249375 (395h43m3.156663002s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:09.469438565 (395h43m9.352852186s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:10.167123637 (395h43m10.050537256s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:38.676415445 (395h43m38.559829065s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:45.960480350 (395h43m45.843894004s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960513865 (395h43m45.843927481s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960590132 (395h43m45.844003749s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960601510 (395h43m45.844015126s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960620902 (395h43m45.844034518s) [seg_pos={0 0}, upload=316447]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:54:46.067458842 (395h43m45.950872454s) [upload=316447]: Done uploading segments... Jul 9 20:54:46.067486298 (395h43m45.950899914s) [upload=316447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.382855846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.308268965s FAIL (2025-07-09 21:25:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.308299679s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 26.81470764s FAIL (2025-07-09 21:25:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.814730859s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 21.513731173s FAIL (2025-07-09 21:25:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ul 9 20:53:05.181732432 (395h42m5.065146066s) [upload=316437]: Upload finished. err: <nil> Jul 9 20:53:46.239759806 (395h42m46.123173434s) [upload=316447]: Starting upload Jul 9 20:53:46.239879159 (395h42m46.123292791s) [upload=316447]: Done starting upload Jul 9 20:53:46.240039985 (395h42m46.123453618s) [upload=316447]: Uploading segments... Jul 9 20:53:46.240311285 (395h42m46.123724915s) [upload=316447]: Got next segment. Inline: false Jul 9 20:53:46.240364403 (395h42m46.123778042s) [upload=316447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:53:46.240374251 (395h42m46.123787889s) [upload=316447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:53:46.440120932 (395h42m46.323534548s) [seg_pos={0 0}, upload=316447]: Begin upload segment... Jul 9 20:53:46.440179950 (395h42m46.323593568s) [seg_pos={0 0}, upload=316447]: Done begin upload segment. Jul 9 20:53:46.440217375 (395h42m46.323631001s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440650993 (395h42m46.32406462s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440924286 (395h42m46.324337911s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441186006 (395h42m46.324599632s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441444482 (395h42m46.324858108s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441690913 (395h42m46.325104542s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442118579 (395h42m46.325532246s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442302181 (395h42m46.32571581s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442705594 (395h42m46.326119222s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443169151 (395h42m46.326582779s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443516420 (395h42m46.326930048s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443951987 (395h42m46.327365616s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444246383 (395h42m46.32766001s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444526835 (395h42m46.327940466s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444714260 (395h42m46.328127876s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445082976 (395h42m46.328496603s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445455193 (395h42m46.328868819s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.446344122 (395h42m46.329757751s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:47.020800550 (395h42m46.90421418s) [upload=316447]: Next returned nil segment Jul 9 20:53:47.020819787 (395h42m46.904233401s) [upload=316447]: Waiting for error group managing segments... Jul 9 20:53:47.156598268 (395h42m47.04001189s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.159389097 (395h42m47.042802716s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.160887797 (395h42m47.044301415s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.163302365 (395h42m47.046715982s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.167069778 (395h42m47.050483398s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.168030609 (395h42m47.051444228s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.383117245 (395h42m47.266530862s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:48.832058921 (395h42m48.71547254s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:49.721740279 (395h42m49.605153896s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: storage node overloaded, request limit: 2000 Jul 9 20:53:58.022618980 (395h42m57.9060326s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:03.273249375 (395h43m3.156663002s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:09.469438565 (395h43m9.352852186s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:10.167123637 (395h43m10.050537256s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:38.676415445 (395h43m38.559829065s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:45.960480350 (395h43m45.843894004s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960513865 (395h43m45.843927481s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960590132 (395h43m45.844003749s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960601510 (395h43m45.844015126s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960620902 (395h43m45.844034518s) [seg_pos={0 0}, upload=316447]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:54:46.067458842 (395h43m45.950872454s) [upload=316447]: Done uploading segments... Jul 9 20:54:46.067486298 (395h43m45.950899914s) [upload=316447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.513851164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 15.031018575s FAIL (2025-07-09 21:24:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.031044138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.357974806s FAIL (2025-07-09 21:24:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.358008796s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.662818157s FAIL (2025-07-09 21:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.662842554s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.379219505s FAIL (2025-07-09 21:24:06 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.379249394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 26.568290481s FAIL (2025-07-09 21:24:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ul 9 20:53:05.181732432 (395h42m5.065146066s) [upload=316437]: Upload finished. err: <nil> Jul 9 20:53:46.239759806 (395h42m46.123173434s) [upload=316447]: Starting upload Jul 9 20:53:46.239879159 (395h42m46.123292791s) [upload=316447]: Done starting upload Jul 9 20:53:46.240039985 (395h42m46.123453618s) [upload=316447]: Uploading segments... Jul 9 20:53:46.240311285 (395h42m46.123724915s) [upload=316447]: Got next segment. Inline: false Jul 9 20:53:46.240364403 (395h42m46.123778042s) [upload=316447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:53:46.240374251 (395h42m46.123787889s) [upload=316447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:53:46.440120932 (395h42m46.323534548s) [seg_pos={0 0}, upload=316447]: Begin upload segment... Jul 9 20:53:46.440179950 (395h42m46.323593568s) [seg_pos={0 0}, upload=316447]: Done begin upload segment. Jul 9 20:53:46.440217375 (395h42m46.323631001s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440650993 (395h42m46.32406462s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440924286 (395h42m46.324337911s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441186006 (395h42m46.324599632s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441444482 (395h42m46.324858108s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441690913 (395h42m46.325104542s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442118579 (395h42m46.325532246s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442302181 (395h42m46.32571581s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442705594 (395h42m46.326119222s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443169151 (395h42m46.326582779s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443516420 (395h42m46.326930048s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443951987 (395h42m46.327365616s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444246383 (395h42m46.32766001s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444526835 (395h42m46.327940466s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444714260 (395h42m46.328127876s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445082976 (395h42m46.328496603s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445455193 (395h42m46.328868819s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.446344122 (395h42m46.329757751s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:47.020800550 (395h42m46.90421418s) [upload=316447]: Next returned nil segment Jul 9 20:53:47.020819787 (395h42m46.904233401s) [upload=316447]: Waiting for error group managing segments... Jul 9 20:53:47.156598268 (395h42m47.04001189s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.159389097 (395h42m47.042802716s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.160887797 (395h42m47.044301415s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.163302365 (395h42m47.046715982s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.167069778 (395h42m47.050483398s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.168030609 (395h42m47.051444228s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.383117245 (395h42m47.266530862s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:48.832058921 (395h42m48.71547254s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:49.721740279 (395h42m49.605153896s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: storage node overloaded, request limit: 2000 Jul 9 20:53:58.022618980 (395h42m57.9060326s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:03.273249375 (395h43m3.156663002s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:09.469438565 (395h43m9.352852186s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:10.167123637 (395h43m10.050537256s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:38.676415445 (395h43m38.559829065s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:45.960480350 (395h43m45.843894004s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960513865 (395h43m45.843927481s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960590132 (395h43m45.844003749s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960601510 (395h43m45.844015126s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960620902 (395h43m45.844034518s) [seg_pos={0 0}, upload=316447]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:54:46.067458842 (395h43m45.950872454s) [upload=316447]: Done uploading segments... Jul 9 20:54:46.067486298 (395h43m45.950899914s) [upload=316447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.568312864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 17.485534706s FAIL (2025-07-09 21:23:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.485564803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.682592189s FAIL (2025-07-09 21:23:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.682629667s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 43.66474983s FAIL (2025-07-09 21:23:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ul 9 20:53:05.181732432 (395h42m5.065146066s) [upload=316437]: Upload finished. err: <nil> Jul 9 20:53:46.239759806 (395h42m46.123173434s) [upload=316447]: Starting upload Jul 9 20:53:46.239879159 (395h42m46.123292791s) [upload=316447]: Done starting upload Jul 9 20:53:46.240039985 (395h42m46.123453618s) [upload=316447]: Uploading segments... Jul 9 20:53:46.240311285 (395h42m46.123724915s) [upload=316447]: Got next segment. Inline: false Jul 9 20:53:46.240364403 (395h42m46.123778042s) [upload=316447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:53:46.240374251 (395h42m46.123787889s) [upload=316447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:53:46.440120932 (395h42m46.323534548s) [seg_pos={0 0}, upload=316447]: Begin upload segment... Jul 9 20:53:46.440179950 (395h42m46.323593568s) [seg_pos={0 0}, upload=316447]: Done begin upload segment. Jul 9 20:53:46.440217375 (395h42m46.323631001s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440650993 (395h42m46.32406462s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.440924286 (395h42m46.324337911s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441186006 (395h42m46.324599632s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441444482 (395h42m46.324858108s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.441690913 (395h42m46.325104542s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442118579 (395h42m46.325532246s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442302181 (395h42m46.32571581s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.442705594 (395h42m46.326119222s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443169151 (395h42m46.326582779s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443516420 (395h42m46.326930048s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.443951987 (395h42m46.327365616s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444246383 (395h42m46.32766001s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444526835 (395h42m46.327940466s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.444714260 (395h42m46.328127876s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445082976 (395h42m46.328496603s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.445455193 (395h42m46.328868819s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:46.446344122 (395h42m46.329757751s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Uploading piece... Jul 9 20:53:47.020800550 (395h42m46.90421418s) [upload=316447]: Next returned nil segment Jul 9 20:53:47.020819787 (395h42m46.904233401s) [upload=316447]: Waiting for error group managing segments... Jul 9 20:53:47.156598268 (395h42m47.04001189s) [noise=true , address=77.90.31.12:4603 , piece_id=RB6XNVBOF373CV2ORP736LESCVTPEJIXEZJQDL3MBIDQFLQ2AXLQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.159389097 (395h42m47.042802716s) [noise=true , address=77.90.31.11:4582 , piece_id=FYEQYVURALHPDG6JUOA2SDXTTVTXPA6CXEJ2X7D7BFS2B7JJKW4Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.160887797 (395h42m47.044301415s) [noise=true , address=77.90.31.18:4824 , piece_id=LLO7JQBAN7BCSNJVXOVSE4YQCBTU74F237MBZRJ6NRBAPAKIBH3Q, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.163302365 (395h42m47.046715982s) [noise=true , address=77.90.31.21:4911 , piece_id=II36UAEZ7F7LTRKKMJPGKEIAR7XLCOQOAGG4CYH3FOZ6VUHVTNVA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.167069778 (395h42m47.050483398s) [noise=true , address=77.90.31.13:4630 , piece_id=L7QDIQGANAS6DNSQCVOAVRI7N74IWUODKS7SF7BLWDEKNAKO6QEA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.168030609 (395h42m47.051444228s) [noise=true , address=77.90.31.15:4702 , piece_id=XSHZPS6CN3AQQZ7GFJVJJM4YFWJODUWEPOT6HKQ3NFSHOBSUPWGA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:47.383117245 (395h42m47.266530862s) [noise=true , address=79.127.220.98:4423 , piece_id=2ZQKHDZEWY5EGUWLRLQBMEUWTWZUO2MK3A5VGUDTTGZDNOBADXPQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:48.832058921 (395h42m48.71547254s) [noise=true , address=79.127.201.215:4348 , piece_id=P643SCY27W5ZOBMVT3LD6LZD5XTP3OW5OYGX5TD32T5JKVXD7HHA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:53:49.721740279 (395h42m49.605153896s) [noise=true , address=45.91.95.2:4002 , piece_id=OLCA5YKYX4AYB3BFE4SY2ZBGQUDVI3WE7R7CDAABMENLFLQL46OQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1ceeydU13HxVgA5trT5MumrQoCYKWURrkoTZLPXBXjGiEC23W9, address:45.91.95.2:4002): protocol: storage node overloaded, request limit: 2000 Jul 9 20:53:58.022618980 (395h42m57.9060326s) [noise=true , address=79.127.219.38:4030 , piece_id=TM344C6YJKYD5JDEHF7CY4HQA4WBEQUH6NSFPRC3PS62EPDRAIJQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:03.273249375 (395h43m3.156663002s) [noise=true , address=104.36.82.18:4387 , piece_id=FHI4YIRE5QX5TGS66YP2HWEVZKYHK4SP6ZBOJLLTP6Z5JLK6SC5A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:09.469438565 (395h43m9.352852186s) [noise=true , address=104.36.82.4:4833 , piece_id=E6NMZGIVIZVQYBABG6IKSZGLJXEYPPGY6OG4U6M56BTAYYXM3CJA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:10.167123637 (395h43m10.050537256s) [noise=true , address=104.36.82.51:4081 , piece_id=YTR54NHEDMDMRCOIR7Z6SLAJNR4LLAT7SD5I6ZJLKC2RTHFLUDYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:38.676415445 (395h43m38.559829065s) [noise=true , address=104.36.82.17:4354 , piece_id=NBPDHYLJJSGMSQQ6X4GWTAB432KK7ZIBG54VCR36IOV53TPPD7YA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: <nil> Jul 9 20:54:45.960480350 (395h43m45.843894004s) [noise=true , address=104.36.82.60:4347 , piece_id=IUFS5CNRLX74PROKKFT3TD7LNLTCAP4SF2FQQU2PWKF7IFKBH47A, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1CbCAPrrYNvhv8129X18ftZmajktekzBFMu5rMLYpHriJbN5PM, address:104.36.82.60:4347): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960513865 (395h43m45.843927481s) [noise=true , address=45.91.95.3:4015 , piece_id=AIKGX3EM5TAUSTSZAOVYYNEJDRKXNMH2GDR55CHTOKQJQK4JYNYA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960590132 (395h43m45.844003749s) [noise=true , address=104.36.82.19:4427 , piece_id=DZBXNLAHIVL7K54ORKHKYSBEMT4YH3D3XRBCMEZUDJRCIFBFDLPA, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960601510 (395h43m45.844015126s) [noise=true , address=45.91.95.4:4036 , piece_id=6FQHHGIXR3ZWG7AO25QWYQJPB77WIJVPUDCMILTYDBSXZ7KDQMEQ, seg_pos={0 0}, upload=316447]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:54:45.960620902 (395h43m45.844034518s) [seg_pos={0 0}, upload=316447]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:54:46.067458842 (395h43m45.950872454s) [upload=316447]: Done uploading segments... Jul 9 20:54:46.067486298 (395h43m45.950899914s) [upload=316447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 43.664789679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.300725275s FAIL (2025-07-09 21:23:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.300756341s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.673418953s FAIL (2025-07-09 21:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.673448721s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 24.713622134s FAIL (2025-07-09 21:23:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.713682242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.373944775s FAIL (2025-07-09 21:22:59 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== g_pos={0 0}, upload=316367]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:47:36.244291365 (395h36m36.127704992s) [upload=316367]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:47:36.244296534 (395h36m36.12771016s) [upload=316367]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:47:36.398064624 (395h36m36.281478263s) [upload=316367]: Done uploading segments... Jul 9 20:47:36.398088829 (395h36m36.281502458s) [upload=316367]: Upload finished. err: <nil> Jul 9 20:48:05.350156635 (395h37m5.233570264s) [upload=316375]: Starting upload Jul 9 20:48:05.350246294 (395h37m5.233659914s) [upload=316375]: Done starting upload Jul 9 20:48:05.350375398 (395h37m5.233789018s) [upload=316375]: Uploading segments... Jul 9 20:48:05.350561373 (395h37m5.233974991s) [upload=316375]: Got next segment. Inline: false Jul 9 20:48:05.350614545 (395h37m5.23402817s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:05.350620566 (395h37m5.234034192s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:05.458460235 (395h37m5.341873852s) [seg_pos={0 0}, upload=316375]: Begin upload segment... Jul 9 20:48:05.458527091 (395h37m5.341940719s) [seg_pos={0 0}, upload=316375]: Done begin upload segment. Jul 9 20:48:05.458556195 (395h37m5.34196983s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458562192 (395h37m5.341975828s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.458933587 (395h37m5.342347216s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459058718 (395h37m5.342472347s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459243031 (395h37m5.34265666s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459440806 (395h37m5.342854435s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459561010 (395h37m5.342974638s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.459964235 (395h37m5.343377871s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460006098 (395h37m5.343419737s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460336395 (395h37m5.343750022s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460346182 (395h37m5.343759815s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460394451 (395h37m5.343808083s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460678150 (395h37m5.344091783s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.460949337 (395h37m5.344362969s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461264345 (395h37m5.344677977s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:05.461558061 (395h37m5.34497169s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Uploading piece... Jul 9 20:48:06.075903844 (395h37m5.95931746s) [upload=316375]: Next returned nil segment Jul 9 20:48:06.075920633 (395h37m5.959334263s) [upload=316375]: Waiting for error group managing segments... Jul 9 20:48:06.193841562 (395h37m6.077255175s) [noise=true , address=104.36.82.12:4150 , piece_id=THNEEDG45WKTOE2UUWNUORE47T6BJUMNIX77CLYIDQCWQCUEWG7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.207913388 (395h37m6.091327015s) [noise=true , address=45.91.116.13:4155 , piece_id=MNGVJN2HXXFHVJ3DAG4UKM6ZUW3FYNS5HCWZFE2ITYIR37LKITYQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.216946791 (395h37m6.100360405s) [noise=true , address=45.91.116.14:4190 , piece_id=HHID2EF5TJQOWIPWVAMWYQ3ZJVWV7QDVVA44G6S4IRSUNZRWILMQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.223899481 (395h37m6.107313111s) [noise=true , address=45.91.116.12:4227 , piece_id=O6PSKTAFNSB7EGS4XYN5JX7FGV7RMNOLVEPLUUBO34OT2BNE3MBQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.224185439 (395h37m6.107599059s) [noise=true , address=104.36.82.10:4078 , piece_id=TWYODK6NJ7NAAUUVTJD7GSO2ZW6ALCZNYMLMWZNM3AXZBRWCIJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.228926526 (395h37m6.112340161s) [noise=true , address=77.90.31.17:4772 , piece_id=2AEBDEJDYC2DLSGEMIYDU7IOBWOSFFGNJD5Z2OWAF7QWB6BB2USA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231175580 (395h37m6.114589217s) [noise=true , address=45.91.116.9:4262 , piece_id=IZHLUIKUZTCNIL2K7V5OCSEIS52STDTXLOMWJIFXHJEX6HBYD7CA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.231929100 (395h37m6.115342718s) [noise=true , address=77.90.31.15:4698 , piece_id=7O2XEHXVN4EPW7OK2IX22MQWYFG25645S5NUQN3NBHW5DPMSX2MA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.234668051 (395h37m6.11808168s) [noise=true , address=77.90.31.11:4553 , piece_id=6BYT5IOCYEYGGDVNODBBLQVFKTOFHZSPV6ZXU2EUY25BF2N2WJFA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.235896082 (395h37m6.119309711s) [noise=true , address=77.90.31.13:4627 , piece_id=ZY7KKGZASUM7HWAMBDCBERUYAYPW6DJFA4VNWN4AXUAFE76XW7WA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:06.245500968 (395h37m6.128914582s) [noise=true , address=104.36.82.11:4115 , piece_id=VI4UWB2ROZGL6P5UZ2ND6MSM3PNRW2OJPVJPXJDVIM3LH55B7R7A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:29.284655472 (395h37m29.168069085s) [noise=true , address=104.36.82.6:4874 , piece_id=7W7GZJLTWU37FH6QNDJMICZ2YNHJ7OVJOEIC2CYO7ID74QOF3U6A, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953757464 (395h37m33.837171094s) [noise=true , address=104.36.82.3:4765 , piece_id=PRPV2SBSURY7V7HRW32BEBQBQ44QVXJC77YRI7N5OEZ2S25T4DQA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: <nil> Jul 9 20:48:33.953797165 (395h37m33.837210795s) [seg_pos={0 0}, upload=316375]: Segment reached optimal threshold of 13 pieces. Jul 9 20:48:33.954000365 (395h37m33.837413985s) [noise=true , address=104.36.82.2:4731 , piece_id=VELITJJTCY2QED7XLBQSYW5JY63LU4WXE25YEGCRNT6Y3PFBQGAQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Jul 9 20:48:33.954159166 (395h37m33.837572794s) [noise=true , address=104.36.82.4:4803 , piece_id=HX2V5HWN4O5QZRAQNHYT5UMXQA57KNAC42EXBWN547GEAUW5KLSQ, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44968->104.36.82.4:4803: use of closed network connection Jul 9 20:48:33.954230334 (395h37m33.837643962s) [noise=true , address=104.36.82.5:4839 , piece_id=3IDN23C3L3AOOM7XZGGKLGPI5PUYCY5VL76JZKKV2FPZRC4VVPJA, seg_pos={0 0}, upload=316375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60818->104.36.82.5:4839: use of closed network connection Jul 9 20:48:33.954257932 (395h37m33.837671545s) [seg_pos={0 0}, upload=316375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 20:48:33.954398670 (395h37m33.8378123s) [upload=316375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 20:48:33.954403994 (395h37m33.837817606s) [upload=316375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 20:48:34.117929330 (395h37m34.001342957s) [upload=316375]: Done uploading segments... Jul 9 20:48:34.117948670 (395h37m34.001362287s) [upload=316375]: Upload finished. err: <nil> Duration: 2.373978085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 16.146955189s FAIL (2025-07-09 21:22:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.146967267s core uplink us-select-1 us1-placement-20

1
...
3562
3563
3564
3565
3566
...
3727