Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.710552593s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:22:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.710587319s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.377307637s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:22:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.377339609s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................XX....................................] duration:
1.709342365s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:22:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.709400279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.698078555s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:22:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.698406833s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.318308596s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:22:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.318336515s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................XX...................................] duration:
1.713117023s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:21:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.713142439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.758065287s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:21:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.758094984s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.421186013s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:21:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.421213583s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................XX..................................] duration:
1.752257346s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:20:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.752285044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.34581196s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:20:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.345838586s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................XX.................................] duration:
2.137041615s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:20:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.137071882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.710824693s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:20:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.710852209s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................XX................................] duration:
2.043424848s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:20:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.043453603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.37843078s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:19:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.378458088s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.719231773s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:19:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.719259397s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................XX...............................] duration:
1.547909169s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:19:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.547969759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.349708064s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:19:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.349729972s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................XX..............................] duration:
1.625011614s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:18:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.625037963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.606563101s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:18:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.606579043s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................XX.............................] duration:
1.627136006s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:18:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.62716746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.353180456s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:18:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.353211973s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.647803895s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:17:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.64823499s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.33964406s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:17:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.339709513s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................XX............................] duration:
1.757413429s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:17:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.757435254s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.2814377s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:17:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.281459894s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.669571438s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:16:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.669663578s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................XX...........................] duration:
1.68418896s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:16:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.684214892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.219611865s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:16:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.219644333s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.593359505s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:16:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.593393191s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.277228789s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:15:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.277261288s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................XX..........................] duration:
1.618471766s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:15:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.618552294s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................XX.........................] duration:
1.649037374s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:15:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.649066996s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.800062297s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:15:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.800089466s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.288151755s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:14:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.288183966s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................XX........................] duration:
1.667398519s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:14:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.66742968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.826167059s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:14:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.826192142s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................XX.......................] duration:
1.69418123s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:13:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.694210596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.297893041s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:13:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.29792286s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.81152907s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:13:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.811556574s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.246473497s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:13:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.246504938s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................XX......................] duration:
2.181912606s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:13:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.181942734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.828450823s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:12:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.828475809s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration:
1.620024231s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:12:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.620055907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.295027625s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:12:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.295053307s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.875592015s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:12:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.875615324s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................XX....................] duration:
1.577188503s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:11:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.577213264s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.264506651s trace:
https://jaeger.dev.storj.io/trace/5174e2e9d0983e17
FAIL (2026-02-09 02:11:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Feb 9 01:55:13.975108561 (585h59m51.859062761s) [upload=515859]: Starting upload
Feb 9 01:55:13.975167828 (585h59m51.859122018s) [upload=515859]: Done starting upload
Feb 9 01:55:13.975255005 (585h59m51.859209198s) [upload=515859]: Uploading segments...
Feb 9 01:55:13.975456860 (585h59m51.85941106s) [upload=515859]: Got next segment. Inline: false
Feb 9 01:55:13.975493721 (585h59m51.859447921s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 9 01:55:13.975498495 (585h59m51.859452693s) [upload=515859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 9 01:55:14.099456994 (585h59m51.983411193s) [seg_pos={0 0}, upload=515859]: Begin upload segment...
Feb 9 01:55:14.099538395 (585h59m51.983492584s) [seg_pos={0 0}, upload=515859]: Done begin upload segment.
Feb 9 01:55:14.099606226 (585h59m51.983560435s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.100678411 (585h59m51.984632618s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101158950 (585h59m51.985113157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.101545765 (585h59m51.985499974s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102048326 (585h59m51.986002536s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102345010 (585h59m51.98629936s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102383053 (585h59m51.986337255s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102767577 (585h59m51.986721785s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.102777165 (585h59m51.986731374s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103116740 (585h59m51.987070944s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103145510 (585h59m51.987099708s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.103598574 (585h59m51.987552776s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104178273 (585h59m51.988132708s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.104933173 (585h59m51.988887384s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.105488383 (585h59m51.989442587s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.106041570 (585h59m51.98999577s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Uploading piece...
Feb 9 01:55:14.711491268 (585h59m52.595445454s) [upload=515859]: Next returned nil segment
Feb 9 01:55:14.711507467 (585h59m52.595461667s) [upload=515859]: Waiting for error group managing segments...
Feb 9 01:55:14.843086256 (585h59m52.727040466s) [noise=true , address=45.91.116.9:4296 , piece_id=IKR3XBD3A4Z5V73AGTNQFEHPTOZICDZQ3XYC6YFBNTUZKCPRP6NQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.843628548 (585h59m52.727582748s) [noise=true , address=45.91.116.4:4947 , piece_id=57JKT7OIX5ZRCMTBGUYDD2QJRTNT6LHHK7DHH2DUTZKBCG5GNQIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.855339458 (585h59m52.739293658s) [noise=true , address=45.91.116.11:4152 , piece_id=IXH64FOMQH362ZIMHKWIZWBKJLT6PNQFBNE7ALO2MIEMBU6YAL4A, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.859836947 (585h59m52.743791152s) [noise=true , address=45.91.116.8:4080 , piece_id=XWQH2K3EJBG25VEUPEVQ5HKKKN4RLQQKB6IX5XZ4LNMWET4YFPSQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:14.873177386 (585h59m52.757131608s) [noise=true , address=45.91.116.7:4024 , piece_id=E2VWMM6YK4T5LKQ6DAMGAP3SXWZXMABAO46KWW7P5CTVH4Y32AQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.466093661 (585h59m53.350047864s) [noise=true , address=79.127.219.38:4820 , piece_id=LB73TPDLNKCNZYU3IBU2FTPKD53N6VNYKYBOWDML3PPC37YXPSLA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:15.862423614 (585h59m53.746377804s) [noise=true , address=79.127.219.41:4927 , piece_id=GYYR2LVKZTXBRMMUOFQS22K2P7XEZV7TRC3UZ6LFEQNQ3IBCPCIQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.002351321 (585h59m53.886305544s) [noise=true , address=79.127.219.46:4894 , piece_id=YQBCS3ULEUGWHGA45L4FJOBVS4Y6EEGGMRF2L7L55FNAUH2BZGWA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.192742299 (585h59m54.076696502s) [noise=true , address=79.127.219.45:4856 , piece_id=NBGDSB6TESLC5HO7H23KZMWFYT2PZ2PXLSFSQ7YBDTGMMB4GN6IA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:16.637273201 (585h59m54.5212274s) [noise=true , address=79.127.219.38:4818 , piece_id=DFGERCRH7FDLGN7DSCRNZOWWTDRJJ72GAUG7AKGHEVLRU5AKM4XA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: <nil>
Feb 9 01:55:43.666964107 (586h0m21.550918312s) [noise=true , address=77.90.31.19:4841 , piece_id=RXZ7JO75NR7V7BSK6GGZHV35UREQHDBDLP7DYJ75LVCLI74BHYYA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:46278->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667140689 (586h0m21.55109489s) [noise=true , address=77.90.31.13:4627 , piece_id=WIDIPWS7BLOJYP4TA3KXMDQEQ7OAIAMNGOXK3FFR4IUO2MYOSBNA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 139.59.144.22:49054->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667278242 (586h0m21.551232445s) [noise=true , address=77.90.31.17:4773 , piece_id=B3VDFXCZIXZ4DYHDRVXHMFLE3HGGULNTZJ2USYKRFTGRBOSX4DQQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: read tcp 139.59.144.22:40242->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667343147 (586h0m21.551297349s) [noise=true , address=77.90.31.14:4661 , piece_id=YH2CS2OFR5GUZMEUEGYEXYJ4NXVCN6DSOTGWYMQIGNHC5EVDORMQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 139.59.144.22:60578->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667395555 (586h0m21.55134976s) [noise=true , address=77.90.31.14:4664 , piece_id=YP662EGW5K7CTKEAGAYZEAFC76N4N55NTV7XZUNMXU3DKBSA5RDQ, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: read tcp 139.59.144.22:54134->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667442490 (586h0m21.551396695s) [noise=true , address=77.90.31.13:4629 , piece_id=PSZAYBNCLGGGLRH26K5QQAOVWBTT7L5VJMXNKTNMRDYXXM5ENDPA, seg_pos={0 0}, upload=515859]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:39040->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 9 01:55:43.667464087 (586h0m21.551418273s) [seg_pos={0 0}, upload=515859]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Feb 9 01:55:43.785159595 (586h0m21.669113803s) [upload=515859]: Done uploading segments...
Feb 9 01:55:43.785215714 (586h0m21.669169924s) [upload=515859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.264537813s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
3.064876196s trace:
https://jaeger.dev.storj.io/trace/2bd884fc977f2556
FAIL (2026-02-09 02:11:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.064904472s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................XX...................] duration:
1.611692743s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:11:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.61171801s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................XX..................] duration:
1.592811524s trace:
https://jaeger.dev.storj.io/trace/337ea6b9f2a3d11b
FAIL (2026-02-09 02:10:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.592904391s
core
uplink
us-select-1
us1-placement-20
|