Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.583510229s FAIL (2025-09-22 05:35:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.583546027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X..........X...X.XX..X.............XX..........] duration: 2.606287469s FAIL (2025-09-22 05:35:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.606343294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.772554328s FAIL (2025-09-22 05:35:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.772590833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X....X..............X....X...XX...XX....XX..............X...] duration: 2.795697355s FAIL (2025-09-22 05:34:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.795934415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X..........X...X.XX..X.............XX.........] duration: 2.465364498s FAIL (2025-09-22 05:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.465422385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X..........X...X.XX..X.............XX........] duration: 2.644558539s FAIL (2025-09-22 05:33:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.644668719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.519753367s FAIL (2025-09-22 05:33:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.519857673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X....X..............X....X...XX...XX....XX..............X..] duration: 2.805501351s FAIL (2025-09-22 05:33:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.805524807s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X..........X...X.XX..X.............XX.......] duration: 2.49530099s FAIL (2025-09-22 05:33:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.495335868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.703003196s FAIL (2025-09-22 05:33:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.70308237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X....X..............X....X...XX...XX....XX..............X.] duration: 3.082512393s FAIL (2025-09-22 05:33:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.082545367s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X....X..............X....X...XX...XX....XX..............X] duration: 30.001397645s FAIL (2025-09-22 05:33:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001423144s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.528848061s FAIL (2025-09-22 05:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.528886216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X..........X...X.XX..X.............XX......] duration: 2.514759932s FAIL (2025-09-22 05:32:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.514800997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.406430735s FAIL (2025-09-22 05:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.406540474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X..........X...X.XX..X.............XX.....] duration: 2.553512434s FAIL (2025-09-22 05:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.553538643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X....X..............X....X...XX...XX....XX..............] duration: 2.919781925s FAIL (2025-09-22 05:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.920386193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X..........X...X.XX..X.............XX....] duration: 2.613281294s FAIL (2025-09-22 05:31:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.613336567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.680446372s FAIL (2025-09-22 05:31:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.680481758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X....X..............X....X...XX...XX....XX.............] duration: 2.926175221s FAIL (2025-09-22 05:30:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.926204282s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.4735657s FAIL (2025-09-22 05:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.473606385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X..........X...X.XX..X.............XX...] duration: 2.354048719s FAIL (2025-09-22 05:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.354081734s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X....X..............X....X...XX...XX....XX............] duration: 3.030335772s FAIL (2025-09-22 05:30:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.030372691s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.665944399s FAIL (2025-09-22 05:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.666025039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......X....X..............X....X...XX...XX....XX...........] duration: 2.707649565s FAIL (2025-09-22 05:29:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707672736s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X..........X...X.XX..X.............XX..] duration: 2.435555522s FAIL (2025-09-22 05:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.435598614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X..........X...X.XX..X.............XX.] duration: 2.649509051s FAIL (2025-09-22 05:29:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.649532277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X..........X...X.XX..X.............XX] duration: 30.115694822s FAIL (2025-09-22 05:29:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 05:28:42.616421096 (567h21m3.108853524s) [upload=499090]: Starting upload Sep 22 05:28:42.616493372 (567h21m3.108925801s) [upload=499090]: Done starting upload Sep 22 05:28:42.616626939 (567h21m3.109059374s) [upload=499090]: Uploading segments... Sep 22 05:28:42.616789227 (567h21m3.109221656s) [upload=499090]: Got next segment. Inline: false Sep 22 05:28:42.616842407 (567h21m3.109274835s) [upload=499090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:28:42.616847304 (567h21m3.109279733s) [upload=499090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:28:42.734578494 (567h21m3.227010927s) [seg_pos={0 0}, upload=499090]: Begin upload segment... Sep 22 05:28:42.734627731 (567h21m3.227060166s) [seg_pos={0 0}, upload=499090]: Done begin upload segment. Sep 22 05:28:42.734656133 (567h21m3.227088577s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.734739268 (567h21m3.22717172s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735039239 (567h21m3.227471691s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735232562 (567h21m3.227665008s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735436717 (567h21m3.227869162s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735529778 (567h21m3.227962223s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735789442 (567h21m3.228221891s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.735796963 (567h21m3.228229408s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736072274 (567h21m3.228504715s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736115262 (567h21m3.228547707s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736355948 (567h21m3.22878839s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736455152 (567h21m3.228887601s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736615345 (567h21m3.22904779s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736745206 (567h21m3.229177651s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.736884511 (567h21m3.229316953s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:42.737028857 (567h21m3.229461304s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Uploading piece... Sep 22 05:28:43.351646202 (567h21m3.844078632s) [upload=499090]: Next returned nil segment Sep 22 05:28:43.351691657 (567h21m3.844124099s) [upload=499090]: Waiting for error group managing segments... Sep 22 05:28:43.482650714 (567h21m3.975083152s) [noise=true , address=77.90.31.12:4592 , piece_id=MQTFKX6NQEIID4OP2GAHKGUFWTCZADKWSUNU43QCUG7QESQW336Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483709593 (567h21m3.976142023s) [noise=true , address=45.91.116.12:4225 , piece_id=FVFLIH24VQMYUFFVCUWLUTMLTETNWQVMVSBTPNWO63MS64J5K4YA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.483851985 (567h21m3.976284435s) [noise=true , address=77.90.31.19:4842 , piece_id=XAZ4FVR737CNGKQJOQRIANC67KM4TFWYI4U2RUEQPMCTVF2OA5FA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.484783735 (567h21m3.977216164s) [noise=true , address=45.91.116.14:4191 , piece_id=PCURGLKL5LRAWSWRBRHCOPKNILNYYEYT6SFAKZGAXADXI2GPL3OQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.487251283 (567h21m3.979683719s) [noise=true , address=77.90.31.14:4661 , piece_id=TCDC7GZJ2WYZGN74X2PQH3JTBUS2WUG64444WQ7HWRRYQPFDGW4Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.493229812 (567h21m3.985662243s) [noise=true , address=77.90.31.11:4554 , piece_id=6QPW2GB4EVZ45IWJATDXRTVJVP253WG7RMJL7HJDINKHIZ5HTOHA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.496771206 (567h21m3.989203636s) [noise=true , address=77.90.31.19:4843 , piece_id=KROSNNS7OZ5Y6KSHJZJJGTCXKIMHTH4K6CDQMGKS6XQMO3CK723Q, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.497241361 (567h21m3.989673793s) [noise=true , address=77.90.31.15:4697 , piece_id=75CAVUSCXXVREUU44LGJQ4FPLBHMSM6ZLGYWR5YNQSBNUED3EQAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.498398236 (567h21m3.990830664s) [noise=true , address=77.90.31.18:4805 , piece_id=ANFK6XCIQ7DRU3OPKD2BIZD5NQ6MQVFYSGB6SYWRAKUIB3422KBQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.501062836 (567h21m3.993495272s) [noise=true , address=45.91.116.10:4083 , piece_id=OHW76WGRUKQUVUVIYMCD5TLX6HZXGIMLXBRUHNGVBHAUHRCGJENQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:28:43.505576879 (567h21m3.998009307s) [noise=true , address=77.90.31.16:4735 , piece_id=752AX4WVHP4H4VXT36T4NPDW7N3HHTKN5KQLUJW7GMCW6DZNC4BQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: <nil> Sep 22 05:29:12.326392877 (567h21m32.818825323s) [noise=true , address=145.79.227.6:4257 , piece_id=QBFADP2OBSDBLXBHFOJIUX4TCZT3V7JV7SEUZD3JGE4REIZST54A, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326708832 (567h21m32.819141285s) [noise=true , address=145.79.227.13:4802 , piece_id=PRMKDO2UCSVHCSZWR7CLQDWMXRRWB6S5ZUX7D6SOQNKMD2H3GGEQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326779669 (567h21m32.819212112s) [noise=true , address=145.79.227.12:4767 , piece_id=5QGZJGBIVDFXC2E7MHNB343ESLATD4J3QDNTE4WA63KD767A3OPA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326821001 (567h21m32.819253453s) [noise=true , address=145.79.227.12:4765 , piece_id=OHCP5GLBUMPLEYCULOCJBGQJL4S7INZDLVGIRFWLBMNRGR6OHXDA, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326846573 (567h21m32.819279023s) [noise=true , address=145.79.227.5:4221 , piece_id=TUK2ZWKEIO55PWLPZ3OM2OLG6LPWA4ZBUGMS6UYF2I6P5ENW5CAQ, seg_pos={0 0}, upload=499090]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:29:12.326900791 (567h21m32.819333244s) [seg_pos={0 0}, upload=499090]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 05:29:12.440214439 (567h21m32.932646869s) [upload=499090]: Done uploading segments... Sep 22 05:29:12.440242159 (567h21m32.93267459s) [upload=499090]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.115721264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.566939817s FAIL (2025-09-22 05:28:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.566979038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX......X....X..............X....X...XX...XX....XX..........] duration: 2.654244411s FAIL (2025-09-22 05:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.654270866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X..........X...X.XX..X.............X] duration: 30.108252138s FAIL (2025-09-22 05:28:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :27:55.116236288 (567h20m15.60866874s) [upload=499079]: Starting upload Sep 22 05:27:55.116317326 (567h20m15.608749772s) [upload=499079]: Done starting upload Sep 22 05:27:55.116436529 (567h20m15.608868963s) [upload=499079]: Uploading segments... Sep 22 05:27:55.116693692 (567h20m15.609126134s) [upload=499079]: Got next segment. Inline: false Sep 22 05:27:55.117388953 (567h20m15.609821398s) [upload=499079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:27:55.117400568 (567h20m15.609833s) [upload=499079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:27:55.236568208 (567h20m15.72900065s) [seg_pos={0 0}, upload=499079]: Begin upload segment... Sep 22 05:27:55.236669841 (567h20m15.729102293s) [seg_pos={0 0}, upload=499079]: Done begin upload segment. Sep 22 05:27:55.236717810 (567h20m15.729150266s) [noise=true , address=77.90.31.10:4520 , piece_id=RPBRGW3VBMYGSRPERD6FWHOMPJGMN5ER3YJ5GWGOLUGPEK7KFJRA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.237299869 (567h20m15.729732318s) [noise=true , address=145.79.227.16:4909 , piece_id=WBFRQCPUDSXJQCZKAWXV54HEPT4MFCXL6HLHFQHTDSFD2ZOEMMZA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.237388642 (567h20m15.729821094s) [noise=true , address=145.79.227.16:4912 , piece_id=22R4XS4JTPGFUUSHE2IUEHOZPY5NCXLK652RU2PST2LYISVSBCHA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.237690572 (567h20m15.730123025s) [noise=true , address=45.91.116.14:4189 , piece_id=HGUA2TYITKH3KE4XWXYFOBKP5NR2EN3HXFFOQ7USLMEV4CLLUB6A, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.237967598 (567h20m15.730400052s) [noise=true , address=45.91.116.11:4119 , piece_id=N336S74J7VFR6TX24PBUY64WPJJQFA66CIWK35U4I2TTS536NAOA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.238090056 (567h20m15.730522509s) [noise=true , address=145.79.227.19:4079 , piece_id=2JURNN44GG664QRMNFESARD3HKNQIMTUMOWNJHFZNCWHQ4MC7YUA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.238373441 (567h20m15.730805891s) [noise=true , address=77.90.31.14:4663 , piece_id=GLOELXX5PQ6QTWR3DLQTINVHRAKCDKX7G3D3INYMMJQXZQIJFWUQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.238444100 (567h20m15.730876554s) [noise=true , address=145.79.227.3:4149 , piece_id=VVG6WQURKYROSVBDHFR35E7PEIUU77UYDZYYJU2UDUAMMTNAGW3Q, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.238767682 (567h20m15.731200177s) [noise=true , address=145.79.227.5:4223 , piece_id=UORSZCUIOMXZWQQ3DC6RFHAU5U6Q6YQGNNMTF75ZVEJDQ7LYRCMQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.238815162 (567h20m15.731247614s) [noise=true , address=145.79.227.11:4731 , piece_id=4LW54B7DHTFDWPWABNH4K3QNLQRLD3RC5VFY3WCR6HPKATJFWVWA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.239456356 (567h20m15.731888862s) [noise=true , address=77.90.31.11:4555 , piece_id=IMZVR2FDYPMESHKFJNFBU7S5UQUIE4KNEOLZGT7HBFADTJZMU73Q, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.239690505 (567h20m15.732122968s) [noise=true , address=45.91.116.14:4191 , piece_id=LVRLKAOTBBJQRFXFFDYLIL4VG6OUQZIV7RGSBFEJVYRHS6GN56XQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.239868973 (567h20m15.732301417s) [noise=true , address=77.90.31.19:4841 , piece_id=W5X3XE3TEG5RHO266UV5BNQAVD2NPZM3OGKTRHCSMUE7VSUA7DJA, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.240237097 (567h20m15.732669547s) [noise=true , address=145.79.227.17:4972 , piece_id=57C6Z7K3KX2KIGNUV55XKJN74YQDG7KDWNULXFD5JQNFA25XV6YQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.240622655 (567h20m15.73305511s) [noise=true , address=145.79.227.15:4875 , piece_id=QASIJMO3ZVID3MPO76T3YHIVNA6Z2RWMXP35RIWT2UNQ62F6KHRQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.241024053 (567h20m15.733456505s) [noise=true , address=45.91.116.8:4047 , piece_id=4AUIOYV6J6O4LXRMAD3LGZNY67OTYU7QJXOIQFYXSGIFNDQROJPQ, seg_pos={0 0}, upload=499079]: Uploading piece... Sep 22 05:27:55.854277814 (567h20m16.346710256s) [upload=499079]: Next returned nil segment Sep 22 05:27:55.854318480 (567h20m16.346750972s) [upload=499079]: Waiting for error group managing segments... Sep 22 05:27:55.985998134 (567h20m16.478430569s) [noise=true , address=45.91.116.8:4047 , piece_id=4AUIOYV6J6O4LXRMAD3LGZNY67OTYU7QJXOIQFYXSGIFNDQROJPQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:55.989237601 (567h20m16.48167003s) [noise=true , address=77.90.31.10:4520 , piece_id=RPBRGW3VBMYGSRPERD6FWHOMPJGMN5ER3YJ5GWGOLUGPEK7KFJRA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:55.989870810 (567h20m16.482303239s) [noise=true , address=45.91.116.14:4191 , piece_id=LVRLKAOTBBJQRFXFFDYLIL4VG6OUQZIV7RGSBFEJVYRHS6GN56XQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:55.990938611 (567h20m16.483371048s) [noise=true , address=77.90.31.11:4555 , piece_id=IMZVR2FDYPMESHKFJNFBU7S5UQUIE4KNEOLZGT7HBFADTJZMU73Q, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:55.993631079 (567h20m16.486063514s) [noise=true , address=45.91.116.14:4189 , piece_id=HGUA2TYITKH3KE4XWXYFOBKP5NR2EN3HXFFOQ7USLMEV4CLLUB6A, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:55.995024896 (567h20m16.487457331s) [noise=true , address=45.91.116.11:4119 , piece_id=N336S74J7VFR6TX24PBUY64WPJJQFA66CIWK35U4I2TTS536NAOA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:56.001825724 (567h20m16.494258167s) [noise=true , address=77.90.31.19:4841 , piece_id=W5X3XE3TEG5RHO266UV5BNQAVD2NPZM3OGKTRHCSMUE7VSUA7DJA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:56.006724285 (567h20m16.499156728s) [noise=true , address=77.90.31.14:4663 , piece_id=GLOELXX5PQ6QTWR3DLQTINVHRAKCDKX7G3D3INYMMJQXZQIJFWUQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: <nil> Sep 22 05:27:58.014262466 (567h20m18.5066949s) [noise=true , address=145.79.227.5:4223 , piece_id=UORSZCUIOMXZWQQ3DC6RFHAU5U6Q6YQGNNMTF75ZVEJDQ7LYRCMQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op Sep 22 05:28:24.826044438 (567h20m45.318476873s) [noise=true , address=145.79.227.16:4909 , piece_id=WBFRQCPUDSXJQCZKAWXV54HEPT4MFCXL6HLHFQHTDSFD2ZOEMMZA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826351096 (567h20m45.31878354s) [noise=true , address=145.79.227.17:4972 , piece_id=57C6Z7K3KX2KIGNUV55XKJN74YQDG7KDWNULXFD5JQNFA25XV6YQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826428031 (567h20m45.31886051s) [noise=true , address=145.79.227.15:4875 , piece_id=QASIJMO3ZVID3MPO76T3YHIVNA6Z2RWMXP35RIWT2UNQ62F6KHRQ, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826485091 (567h20m45.318917527s) [noise=true , address=145.79.227.19:4079 , piece_id=2JURNN44GG664QRMNFESARD3HKNQIMTUMOWNJHFZNCWHQ4MC7YUA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826575880 (567h20m45.319008316s) [noise=true , address=145.79.227.3:4149 , piece_id=VVG6WQURKYROSVBDHFR35E7PEIUU77UYDZYYJU2UDUAMMTNAGW3Q, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826643278 (567h20m45.319075721s) [noise=true , address=145.79.227.16:4912 , piece_id=22R4XS4JTPGFUUSHE2IUEHOZPY5NCXLK652RU2PST2LYISVSBCHA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:28:24.826664618 (567h20m45.319097057s) [noise=true , address=145.79.227.11:4731 , piece_id=4LW54B7DHTFDWPWABNH4K3QNLQRLD3RC5VFY3WCR6HPKATJFWVWA, seg_pos={0 0}, upload=499079]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 05:28:24.826706915 (567h20m45.319139368s) [seg_pos={0 0}, upload=499079]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:28:24.932968679 (567h20m45.425401123s) [upload=499079]: Done uploading segments... Sep 22 05:28:24.933001970 (567h20m45.425434421s) [upload=499079]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.108283249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.508413259s FAIL (2025-09-22 05:28:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.50844687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX......X....X..............X....X...XX...XX....XX.........] duration: 2.766444799s FAIL (2025-09-22 05:27:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.766480812s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX......X....X..............X....X...XX...XX....XX........] duration: 2.653145118s FAIL (2025-09-22 05:27:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.653381488s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..........X...X.XX..X.............] duration: 2.522284866s FAIL (2025-09-22 05:27:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:59.679315241 (567h10m20.171747669s) [upload=498936]: Starting upload Sep 22 05:17:59.679411334 (567h10m20.171843786s) [upload=498936]: Done starting upload Sep 22 05:17:59.679716319 (567h10m20.172148755s) [upload=498936]: Uploading segments... Sep 22 05:17:59.679829213 (567h10m20.172261663s) [upload=498936]: Got next segment. Inline: false Sep 22 05:17:59.679950899 (567h10m20.172383345s) [upload=498936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:59.679962985 (567h10m20.172395637s) [upload=498936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:59.800475798 (567h10m20.292908243s) [seg_pos={0 0}, upload=498936]: Begin upload segment... Sep 22 05:17:59.800546024 (567h10m20.292978468s) [seg_pos={0 0}, upload=498936]: Done begin upload segment. Sep 22 05:17:59.800769539 (567h10m20.293201981s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.800893277 (567h10m20.293325723s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.801326339 (567h10m20.293758788s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802455300 (567h10m20.294887749s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802758175 (567h10m20.29519062s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803082368 (567h10m20.295514813s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803319641 (567h10m20.295752094s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803736751 (567h10m20.296169199s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803920030 (567h10m20.29635248s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804269267 (567h10m20.296701713s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804375095 (567h10m20.296807545s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804824050 (567h10m20.297256506s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804984620 (567h10m20.297417069s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805400276 (567h10m20.297832723s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805867188 (567h10m20.298299641s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.806095441 (567h10m20.298527898s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:18:00.412209571 (567h10m20.904642008s) [upload=498936]: Next returned nil segment Sep 22 05:18:00.412229059 (567h10m20.904661494s) [upload=498936]: Waiting for error group managing segments... Sep 22 05:18:00.545283219 (567h10m21.037715658s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.553265448 (567h10m21.045697895s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.554195368 (567h10m21.046627813s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.555727408 (567h10m21.04815984s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.566137295 (567h10m21.05856974s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.576185664 (567h10m21.068618112s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579252726 (567h10m21.071685161s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579550122 (567h10m21.071982554s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.580613889 (567h10m21.073046322s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.583606750 (567h10m21.076039182s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.584291982 (567h10m21.076724415s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.610029812 (567h10m21.102462242s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:02.263907796 (567h10m22.756340226s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370137159 (567h10m49.862569603s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370226393 (567h10m49.862658847s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370259958 (567h10m49.862692512s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370300577 (567h10m49.862733023s) [seg_pos={0 0}, upload=498936]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:29.484566976 (567h10m49.976999411s) [upload=498936]: Done uploading segments... Sep 22 05:18:29.484595671 (567h10m49.977028104s) [upload=498936]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.522319797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.424951239s FAIL (2025-09-22 05:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.424988842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX......X....X..............X....X...XX...XX....XX.......] duration: 2.730177589s FAIL (2025-09-22 05:26:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.73019983s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X..........X...X.XX..X............] duration: 2.408155931s FAIL (2025-09-22 05:26:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:59.679315241 (567h10m20.171747669s) [upload=498936]: Starting upload Sep 22 05:17:59.679411334 (567h10m20.171843786s) [upload=498936]: Done starting upload Sep 22 05:17:59.679716319 (567h10m20.172148755s) [upload=498936]: Uploading segments... Sep 22 05:17:59.679829213 (567h10m20.172261663s) [upload=498936]: Got next segment. Inline: false Sep 22 05:17:59.679950899 (567h10m20.172383345s) [upload=498936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:59.679962985 (567h10m20.172395637s) [upload=498936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:59.800475798 (567h10m20.292908243s) [seg_pos={0 0}, upload=498936]: Begin upload segment... Sep 22 05:17:59.800546024 (567h10m20.292978468s) [seg_pos={0 0}, upload=498936]: Done begin upload segment. Sep 22 05:17:59.800769539 (567h10m20.293201981s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.800893277 (567h10m20.293325723s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.801326339 (567h10m20.293758788s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802455300 (567h10m20.294887749s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802758175 (567h10m20.29519062s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803082368 (567h10m20.295514813s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803319641 (567h10m20.295752094s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803736751 (567h10m20.296169199s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803920030 (567h10m20.29635248s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804269267 (567h10m20.296701713s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804375095 (567h10m20.296807545s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804824050 (567h10m20.297256506s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804984620 (567h10m20.297417069s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805400276 (567h10m20.297832723s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805867188 (567h10m20.298299641s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.806095441 (567h10m20.298527898s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:18:00.412209571 (567h10m20.904642008s) [upload=498936]: Next returned nil segment Sep 22 05:18:00.412229059 (567h10m20.904661494s) [upload=498936]: Waiting for error group managing segments... Sep 22 05:18:00.545283219 (567h10m21.037715658s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.553265448 (567h10m21.045697895s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.554195368 (567h10m21.046627813s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.555727408 (567h10m21.04815984s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.566137295 (567h10m21.05856974s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.576185664 (567h10m21.068618112s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579252726 (567h10m21.071685161s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579550122 (567h10m21.071982554s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.580613889 (567h10m21.073046322s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.583606750 (567h10m21.076039182s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.584291982 (567h10m21.076724415s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.610029812 (567h10m21.102462242s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:02.263907796 (567h10m22.756340226s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370137159 (567h10m49.862569603s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370226393 (567h10m49.862658847s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370259958 (567h10m49.862692512s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370300577 (567h10m49.862733023s) [seg_pos={0 0}, upload=498936]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:29.484566976 (567h10m49.976999411s) [upload=498936]: Done uploading segments... Sep 22 05:18:29.484595671 (567h10m49.977028104s) [upload=498936]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.408189887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.530678645s FAIL (2025-09-22 05:26:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.530717031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.58839954s FAIL (2025-09-22 05:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.588451491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX......X....X..............X....X...XX...XX....XX......] duration: 2.789201597s FAIL (2025-09-22 05:25:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789216224s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X..........X...X.XX..X...........] duration: 2.366566005s FAIL (2025-09-22 05:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:59.679315241 (567h10m20.171747669s) [upload=498936]: Starting upload Sep 22 05:17:59.679411334 (567h10m20.171843786s) [upload=498936]: Done starting upload Sep 22 05:17:59.679716319 (567h10m20.172148755s) [upload=498936]: Uploading segments... Sep 22 05:17:59.679829213 (567h10m20.172261663s) [upload=498936]: Got next segment. Inline: false Sep 22 05:17:59.679950899 (567h10m20.172383345s) [upload=498936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:59.679962985 (567h10m20.172395637s) [upload=498936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:59.800475798 (567h10m20.292908243s) [seg_pos={0 0}, upload=498936]: Begin upload segment... Sep 22 05:17:59.800546024 (567h10m20.292978468s) [seg_pos={0 0}, upload=498936]: Done begin upload segment. Sep 22 05:17:59.800769539 (567h10m20.293201981s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.800893277 (567h10m20.293325723s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.801326339 (567h10m20.293758788s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802455300 (567h10m20.294887749s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802758175 (567h10m20.29519062s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803082368 (567h10m20.295514813s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803319641 (567h10m20.295752094s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803736751 (567h10m20.296169199s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803920030 (567h10m20.29635248s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804269267 (567h10m20.296701713s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804375095 (567h10m20.296807545s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804824050 (567h10m20.297256506s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804984620 (567h10m20.297417069s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805400276 (567h10m20.297832723s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805867188 (567h10m20.298299641s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.806095441 (567h10m20.298527898s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:18:00.412209571 (567h10m20.904642008s) [upload=498936]: Next returned nil segment Sep 22 05:18:00.412229059 (567h10m20.904661494s) [upload=498936]: Waiting for error group managing segments... Sep 22 05:18:00.545283219 (567h10m21.037715658s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.553265448 (567h10m21.045697895s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.554195368 (567h10m21.046627813s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.555727408 (567h10m21.04815984s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.566137295 (567h10m21.05856974s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.576185664 (567h10m21.068618112s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579252726 (567h10m21.071685161s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579550122 (567h10m21.071982554s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.580613889 (567h10m21.073046322s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.583606750 (567h10m21.076039182s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.584291982 (567h10m21.076724415s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.610029812 (567h10m21.102462242s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:02.263907796 (567h10m22.756340226s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370137159 (567h10m49.862569603s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370226393 (567h10m49.862658847s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370259958 (567h10m49.862692512s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370300577 (567h10m49.862733023s) [seg_pos={0 0}, upload=498936]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:29.484566976 (567h10m49.976999411s) [upload=498936]: Done uploading segments... Sep 22 05:18:29.484595671 (567h10m49.977028104s) [upload=498936]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.366603386s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.XX......X....X..............X....X...XX...XX....XX.....] duration: 2.781546797s FAIL (2025-09-22 05:24:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781573726s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X..........X...X.XX..X..........] duration: 2.630945629s FAIL (2025-09-22 05:24:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:59.679315241 (567h10m20.171747669s) [upload=498936]: Starting upload Sep 22 05:17:59.679411334 (567h10m20.171843786s) [upload=498936]: Done starting upload Sep 22 05:17:59.679716319 (567h10m20.172148755s) [upload=498936]: Uploading segments... Sep 22 05:17:59.679829213 (567h10m20.172261663s) [upload=498936]: Got next segment. Inline: false Sep 22 05:17:59.679950899 (567h10m20.172383345s) [upload=498936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:59.679962985 (567h10m20.172395637s) [upload=498936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:59.800475798 (567h10m20.292908243s) [seg_pos={0 0}, upload=498936]: Begin upload segment... Sep 22 05:17:59.800546024 (567h10m20.292978468s) [seg_pos={0 0}, upload=498936]: Done begin upload segment. Sep 22 05:17:59.800769539 (567h10m20.293201981s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.800893277 (567h10m20.293325723s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.801326339 (567h10m20.293758788s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802455300 (567h10m20.294887749s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802758175 (567h10m20.29519062s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803082368 (567h10m20.295514813s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803319641 (567h10m20.295752094s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803736751 (567h10m20.296169199s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803920030 (567h10m20.29635248s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804269267 (567h10m20.296701713s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804375095 (567h10m20.296807545s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804824050 (567h10m20.297256506s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804984620 (567h10m20.297417069s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805400276 (567h10m20.297832723s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805867188 (567h10m20.298299641s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.806095441 (567h10m20.298527898s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:18:00.412209571 (567h10m20.904642008s) [upload=498936]: Next returned nil segment Sep 22 05:18:00.412229059 (567h10m20.904661494s) [upload=498936]: Waiting for error group managing segments... Sep 22 05:18:00.545283219 (567h10m21.037715658s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.553265448 (567h10m21.045697895s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.554195368 (567h10m21.046627813s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.555727408 (567h10m21.04815984s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.566137295 (567h10m21.05856974s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.576185664 (567h10m21.068618112s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579252726 (567h10m21.071685161s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579550122 (567h10m21.071982554s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.580613889 (567h10m21.073046322s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.583606750 (567h10m21.076039182s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.584291982 (567h10m21.076724415s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.610029812 (567h10m21.102462242s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:02.263907796 (567h10m22.756340226s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370137159 (567h10m49.862569603s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370226393 (567h10m49.862658847s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370259958 (567h10m49.862692512s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370300577 (567h10m49.862733023s) [seg_pos={0 0}, upload=498936]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:29.484566976 (567h10m49.976999411s) [upload=498936]: Done uploading segments... Sep 22 05:18:29.484595671 (567h10m49.977028104s) [upload=498936]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.630981157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.503466356s FAIL (2025-09-22 05:24:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.503500749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.720391121s FAIL (2025-09-22 05:24:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.72045132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.XX......X....X..............X....X...XX...XX....XX....] duration: 2.625345837s FAIL (2025-09-22 05:24:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.625376621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.517404753s FAIL (2025-09-22 05:24:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ObjectBegin Sep 22 05:05:34.373794494 (566h57m54.866226936s) [upload=498759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:05:34.591982663 (566h57m55.08441511s) [seg_pos={0 0}, upload=498759]: Begin upload segment... Sep 22 05:05:34.592084556 (566h57m55.084517052s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593039674 (566h57m55.08547212s) [seg_pos={0 0}, upload=498759]: Done begin upload segment. Sep 22 05:05:34.593196469 (566h57m55.085628898s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.593794312 (566h57m55.086226761s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594151688 (566h57m55.08658414s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594340355 (566h57m55.086772784s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594638370 (566h57m55.087070816s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.594986459 (566h57m55.087418905s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595499345 (566h57m55.087931787s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595520833 (566h57m55.087953288s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595876308 (566h57m55.088308759s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.595908253 (566h57m55.088340702s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596292954 (566h57m55.088725405s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.596711194 (566h57m55.089143643s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597080427 (566h57m55.089512869s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597430372 (566h57m55.089862805s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.597617754 (566h57m55.090050202s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598038150 (566h57m55.090470595s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:34.598361627 (566h57m55.090794076s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Uploading piece... Sep 22 05:05:35.164291673 (566h57m55.656724108s) [upload=498759]: Next returned nil segment Sep 22 05:05:35.164308803 (566h57m55.656741244s) [upload=498759]: Waiting for error group managing segments... Sep 22 05:05:35.301302795 (566h57m55.793735235s) [noise=true , address=77.90.31.11:4579 , piece_id=ESUUKRMP7W7LOUKXBTGLBH24VTBSJQNRVG45M5CDXEVSN6WSGBKQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.301741421 (566h57m55.794173858s) [noise=true , address=45.91.116.14:4217 , piece_id=2STKYM3XB2GQCHDR5Y3UCH7RZSP2S3D4EGJ7RHXW6ZCLDMJ7TCFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.304878741 (566h57m55.797311176s) [noise=true , address=77.90.31.19:4875 , piece_id=KLXOEJRJ3NC6W4ALCKHCAF2ON4L5XALJ7VKXNA7IIHJARRIX7UFA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.305689145 (566h57m55.79812158s) [noise=true , address=77.90.31.21:4903 , piece_id=T3LHN4FLWKZ34RCZMILDXJWRJRGLJANKGJKCPATMRTX6DUBJWWMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.306847119 (566h57m55.799279561s) [noise=true , address=77.90.31.15:4732 , piece_id=UUY4BGXEWBOLJJ5PNRCZRR6O256XGHMMLSMCNNK7XFOO7QY2X5VA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.339666382 (566h57m55.832098823s) [noise=true , address=145.79.227.46:4608 , piece_id=6ICK2FMUVZW73Y7VNRJ7QM6GSD5VVVCRHOSBT4SZ23TPRZOB5MTQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.347651366 (566h57m55.84008381s) [noise=true , address=145.79.227.46:4621 , piece_id=QCMA65ZBY2VLL3WIDZZZBGSSNJZCAV4CBDL7BUNGRP4TUFJG6X5A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.354716468 (566h57m55.847148904s) [noise=true , address=145.79.227.39:4048 , piece_id=5UGUP5GNOHRPXDREC365MI24LKJDALRSZW6IW3TXQRYAUPPXJOAA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.355447946 (566h57m55.847880382s) [noise=true , address=145.79.227.47:4286 , piece_id=P3XQ3X67SNUIJ73JRPQ36AW36G4CJYCGO6KXU5K3VLXJF6JIZKKA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.364679620 (566h57m55.857112104s) [noise=true , address=145.79.227.40:4082 , piece_id=HKJTQJUSTMKP3YKILUCGM5LMOQKWKTNVNAIUX57LEDN3PHHNBCMQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.371887700 (566h57m55.864320128s) [noise=true , address=145.79.227.49:4340 , piece_id=RZCGUPCF3IYA7XHJV2W67QCTXUXIU7JPB3V5SZ53CEEZAKG5YWDA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:35.847984474 (566h57m56.340416916s) [noise=true , address=66.23.193.175:4555 , piece_id=3WWBSWVBWKORPJ4LEITMLLTOUC437H4J3WDZC3G5TFBAP4SDHTTA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: <nil> Sep 22 05:05:36.462717192 (566h57m56.955149625s) [noise=true , address=145.79.227.11:4757 , piece_id=XQNYJQYY6KPLPLGRB2E2XW2RCUHIKYJKSXMHGS6MCGHS4CKBLT3A, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:145.79.227.11:4757): protocol: rpc: dial tcp 145.79.227.11:4757: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068422298 (566h58m54.560854733s) [noise=true , address=145.79.227.14:4867 , piece_id=2YHOBYITZCAT4ZM3SLIMG73VUHJDKPHUTBAGN4GJYEONHJOCN5GQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: rpc: dial tcp 145.79.227.14:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068467561 (566h58m54.560899994s) [noise=true , address=145.79.227.8:4362 , piece_id=G7YEUATSCT6IA5I7HUKTVPKMN47PTXWGDLFXJWLZAEFKWLU6QWNQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:145.79.227.8:4362): protocol: rpc: dial tcp 145.79.227.8:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068538402 (566h58m54.560970833s) [noise=true , address=145.79.227.18:4016 , piece_id=RK2WJCL5M6JXFKO6GIVG6FUB6DRC67UV5HAZUU4V4TB7OSFYTUIQ, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: rpc: dial tcp 145.79.227.18:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068585810 (566h58m54.561018241s) [noise=true , address=145.79.227.13:4815 , piece_id=Z46IWJJTXA6ZJQ2JKRJKCKL3X56I4EXDO6LFO35JWJXJRKMFZTJA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:145.79.227.13:4815): protocol: rpc: dial tcp 145.79.227.13:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068608648 (566h58m54.561041081s) [noise=true , address=145.79.227.10:4433 , piece_id=HNUJP4XBDCZETC2D5AFVUOJKALUNEADYPOJ33JN2JUQKYCQI5VSA, seg_pos={0 0}, upload=498759]: Done uploading piece. err: ecclient: upload failed (node:1VWA9MZwAMNCVwSpoz147FAJRJqP8oVjpqLXtd3oBiyC7t4jsL, address:145.79.227.10:4433): protocol: rpc: dial tcp 145.79.227.10:4433: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:06:34.068628516 (566h58m54.561060949s) [seg_pos={0 0}, upload=498759]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 22 05:06:34.184094759 (566h58m54.676527195s) [upload=498759]: Done uploading segments... Sep 22 05:06:34.184123987 (566h58m54.676556429s) [upload=498759]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.517438545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.XX......X....X..............X....X...XX...XX....XX...] duration: 2.927258632s FAIL (2025-09-22 05:23:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.927296428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X..........X...X.XX..X.........] duration: 2.630586272s FAIL (2025-09-22 05:23:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:17:59.679315241 (567h10m20.171747669s) [upload=498936]: Starting upload Sep 22 05:17:59.679411334 (567h10m20.171843786s) [upload=498936]: Done starting upload Sep 22 05:17:59.679716319 (567h10m20.172148755s) [upload=498936]: Uploading segments... Sep 22 05:17:59.679829213 (567h10m20.172261663s) [upload=498936]: Got next segment. Inline: false Sep 22 05:17:59.679950899 (567h10m20.172383345s) [upload=498936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:17:59.679962985 (567h10m20.172395637s) [upload=498936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:17:59.800475798 (567h10m20.292908243s) [seg_pos={0 0}, upload=498936]: Begin upload segment... Sep 22 05:17:59.800546024 (567h10m20.292978468s) [seg_pos={0 0}, upload=498936]: Done begin upload segment. Sep 22 05:17:59.800769539 (567h10m20.293201981s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.800893277 (567h10m20.293325723s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.801326339 (567h10m20.293758788s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802455300 (567h10m20.294887749s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.802758175 (567h10m20.29519062s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803082368 (567h10m20.295514813s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803319641 (567h10m20.295752094s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803736751 (567h10m20.296169199s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.803920030 (567h10m20.29635248s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804269267 (567h10m20.296701713s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804375095 (567h10m20.296807545s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804824050 (567h10m20.297256506s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.804984620 (567h10m20.297417069s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805400276 (567h10m20.297832723s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.805867188 (567h10m20.298299641s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:17:59.806095441 (567h10m20.298527898s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Uploading piece... Sep 22 05:18:00.412209571 (567h10m20.904642008s) [upload=498936]: Next returned nil segment Sep 22 05:18:00.412229059 (567h10m20.904661494s) [upload=498936]: Waiting for error group managing segments... Sep 22 05:18:00.545283219 (567h10m21.037715658s) [noise=true , address=45.91.116.10:4083 , piece_id=OMOQR5SDEXFDFUVALXACOQXOJGJOZDZL4XDT5PDGAUTJ4XJEDZWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.553265448 (567h10m21.045697895s) [noise=true , address=77.90.31.21:4879 , piece_id=TYGGGJWIDM5M5JYA2SKDJEMQOCQHL6OME3CSFLY3HADHV4MYO3WQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.554195368 (567h10m21.046627813s) [noise=true , address=45.91.116.7:4011 , piece_id=6DNA4SQ7HGJKVESDCFYN3WSDSQ2SP6CNJK6GZ6QREF4XOVTH4KWA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.555727408 (567h10m21.04815984s) [noise=true , address=45.91.116.4:4939 , piece_id=VD6PWIP25RFEUTU7N5TEJ7IBPKCCRG2RCMYVHUMIXPZJSJIW276A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.566137295 (567h10m21.05856974s) [noise=true , address=45.91.116.9:4262 , piece_id=VNP6XCBTUSFNSMILQUKODJMKKLFVX32FIN6H7NHWXPDON4M6SP3A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.576185664 (567h10m21.068618112s) [noise=true , address=77.90.31.15:4697 , piece_id=X7JLBHU4ZHFVHTWH3RKRBCN7AYRHU6I7DWYW7FZ6FKZHIVKSMNQA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579252726 (567h10m21.071685161s) [noise=true , address=45.91.116.5:4299 , piece_id=HAIBFEWPH3G2ZBEOVC6QT7GL4BBL3VBBZK4WQYLSQCOUNSCRFCCA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.579550122 (567h10m21.071982554s) [noise=true , address=77.90.31.19:4843 , piece_id=RGCNJNM2KLFJZ5WGPVTFUQ6UI7XRD677XFC4FMKJO4GJXRCDFUBA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.580613889 (567h10m21.073046322s) [noise=true , address=45.91.116.14:4190 , piece_id=BFBS3SISJGDT4DTWOXPO74KVBSO4MOEAB4HBESTHKN3RDWWSXPOQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.583606750 (567h10m21.076039182s) [noise=true , address=45.91.116.4:4940 , piece_id=HKBSDLNKF34EDIIRZ3VQP6V4F3GKW6N2PKMJWAZQVAWZOCYBDSIQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.584291982 (567h10m21.076724415s) [noise=true , address=77.90.31.12:4592 , piece_id=J2KPGJK6HPACQ7PK56EOYI7XW3U2ATVW32ZYTKI7AZV5XVRHG5CA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:00.610029812 (567h10m21.102462242s) [noise=true , address=45.91.116.11:4119 , piece_id=DG2ZFOJ2TENUA6FQOMK7MVJJHGWNWPMYUN6ZQ2TMMKYWUYVQBHUQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: <nil> Sep 22 05:18:02.263907796 (567h10m22.756340226s) [noise=true , address=145.79.227.16:4909 , piece_id=QRRSD42WLSTXJA6MJIFQM7CC2RM7MOMJXXQYYFBX6EAALNDFMWVA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370137159 (567h10m49.862569603s) [noise=true , address=145.79.227.13:4802 , piece_id=YHOT5235WSCCSERI24VLPZSZTHW4PCNESKRJXHS4MKR7JZ4BD4VQ, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: rpc: dial tcp 145.79.227.13:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370226393 (567h10m49.862658847s) [noise=true , address=145.79.227.6:4259 , piece_id=NWUDFUQDOA2ELC4RWKCLTUFS53ZLVEURNDBWQSHB76DVJ5YHXTKA, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370259958 (567h10m49.862692512s) [noise=true , address=145.79.227.18:4005 , piece_id=D6PS2JVMRDFEDTAB26SQ36UWPRYHYT2XEYXHUKDNAPQTSUI2DV2A, seg_pos={0 0}, upload=498936]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:18:29.370300577 (567h10m49.862733023s) [seg_pos={0 0}, upload=498936]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:18:29.484566976 (567h10m49.976999411s) [upload=498936]: Done uploading segments... Sep 22 05:18:29.484595671 (567h10m49.977028104s) [upload=498936]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.630619147s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)