Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X........X..] duration:
2.880948685s
FAIL (2025-10-06 23:00:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.880984169s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
3.16330833s
FAIL (2025-10-06 23:00:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.163339558s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration:
10.392096206s
FAIL (2025-10-06 23:00:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.392128906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.........X.] duration:
2.541864929s
FAIL (2025-10-06 23:00:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.541902031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.........X] duration:
1m0.118973331s
FAIL (2025-10-06 23:00:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
m12.051558028s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.248295297 (321h29m12.051759897s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249271467 (321h29m12.052736069s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.249537353 (321h29m12.053001945s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250112357 (321h29m12.053576956s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250417134 (321h29m12.053881737s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250657516 (321h29m12.054122112s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.250957826 (321h29m12.054422433s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251190960 (321h29m12.054655555s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.251821271 (321h29m12.055285871s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.253336962 (321h29m12.056801566s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Uploading piece...
Oct 6 22:59:24.831942906 (321h29m12.635407485s) [upload=282911]: Next returned nil segment
Oct 6 22:59:24.831960362 (321h29m12.635424956s) [upload=282911]: Waiting for error group managing segments...
Oct 6 22:59:25.014561142 (321h29m12.818025734s) [noise=true , address=145.79.227.23:4562 , piece_id=OVNWUFCP4VKJLD5FPD24C7OISF7VOBNU3H4LELO25RCM2C5QGJIQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.025424317 (321h29m12.828888911s) [noise=true , address=145.79.227.39:4018 , piece_id=B6DLRTXKCLITTZ2AQO5TRUABUK2TH3J43K77LHSH26LUOQD7MWQQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.066735306 (321h29m12.870199898s) [noise=true , address=145.79.227.30:4784 , piece_id=XLB6V3VELTPPGNT2DPO364YVME2MLBAIQIV3CEEC7GBCH3KXSV4A, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 22:59:25.101662779 (321h29m12.905127363s) [noise=true , address=145.79.227.32:4864 , piece_id=ICT5QSD4OFVLNNJORPJUOTDZQ4Z4WTCXOIMHR5I6VARBZTMK2BXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:17.571870394 (321h30m5.375334985s) [noise=true , address=77.90.31.19:4871 , piece_id=JI63LCXG2BHAPINKCNZ3VNWR5KSXEWRX32RITFEK27HUYWWJ7MDA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:22.271026574 (321h30m10.07449117s) [noise=true , address=77.90.31.14:4666 , piece_id=QKK3OYJ5SQZBCSKRREUWT5RURTP42RKBDC46O2YDJU5EXRSFUYTQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: <nil>
Oct 6 23:00:23.793562436 (321h30m11.59702703s) [noise=true , address=77.90.31.15:4702 , piece_id=LGMCE6PP6AHHW4WUJFZ5ES7IGCPYAHASRAVWJMKYP4V3OJQPTLHA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42162->77.90.31.15:4702: use of closed network connection
Oct 6 23:00:23.793610979 (321h30m11.597075579s) [noise=true , address=45.91.116.11:4120 , piece_id=UMJUJWETDHKQOCVK7PBEAFMCJKJ57VTK55MT3NYQYA5H3CWX5MXA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, address:45.91.116.11:4120): protocol: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35190->45.91.116.11:4120: use of closed network connection
Oct 6 23:00:23.793846256 (321h30m11.597310852s) [noise=true , address=77.90.31.17:4790 , piece_id=JEMWDL6WHNOPB4HHATVC4DVCOC3ALP7OLNLCSNNI25KAJB2XDTBQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 139.59.144.22:55826->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.793891070 (321h30m11.59735567s) [noise=true , address=45.91.116.14:4195 , piece_id=M27EYSLM5UD7T2CRMOWMFG5SXKUVRU3PB7YT2OMHFRAP5Z5KW5BA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794117656 (321h30m11.597582253s) [noise=true , address=77.90.31.16:4738 , piece_id=U4ILYBFLXPMA3JTOJ3GIKG2VRHKN3WKDVFGPQ7A2CRI3H4CCW5QQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: rpc: dial tcp 77.90.31.16:4738: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794273303 (321h30m11.597737898s) [noise=true , address=77.90.31.13:4631 , piece_id=YMUJA3A2XK4XM47C7CXAKMK7FVHOLHRHUZIMHZ227CHIJWSMBYAA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: rpc: dial tcp 77.90.31.13:4631: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794343426 (321h30m11.597808023s) [noise=true , address=45.91.116.5:4316 , piece_id=QIUBHV6PRZBYLMVUT74GSM3INGEUNEQR6KN2EFYQXM6RI3KTMCRQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:59526->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794628089 (321h30m11.598092683s) [noise=true , address=77.90.31.10:4529 , piece_id=OT566DKLB34CJW2V5VJVX2HI6QXOHFYICGU64IQ4ZRLBGDSOQYSA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: rpc: dial tcp 77.90.31.10:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794650454 (321h30m11.598115055s) [noise=true , address=45.91.116.6:4007 , piece_id=CA7D3OTYPKQBUWZDR7VDGUU5CIEBQ5YRAAT6ZZYUS733QAPKJEDQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: read tcp 139.59.144.22:60928->45.91.116.6:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794934492 (321h30m11.598399117s) [noise=true , address=77.90.31.21:4891 , piece_id=IKUEQROHOKIPDTIU6D5G4VFOYZQX5MTJSASGVPAK55YPRDG6SASA, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 139.59.144.22:55224->77.90.31.21:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.794963283 (321h30m11.598427883s) [noise=true , address=45.91.116.12:4249 , piece_id=WDYFZOIQGQ5PN5434UBQ5EQYGKBJAXGKOANT4TUAKYYLPUPWBLEQ, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA, address:45.91.116.12:4249): protocol: read tcp 139.59.144.22:52560->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:23.795114393 (321h30m11.598578986s) [noise=true , address=45.91.116.7:4018 , piece_id=F3ZHBF2E234TNCOZSBHEN6NKNVQFRWKYHXQR2JOZDKKJNKG46L5Q, seg_pos={0 0}, upload=282911]: Done uploading piece. err: ecclient: upload failed (node:12ELnn3hruPY9FSiYuYSz2bNeFRT9Lbu6nYDQEhqRsZGMZD2E9X, address:45.91.116.7:4018): protocol: read tcp 139.59.144.22:33630->45.91.116.7:4018: use of closed network connection; context deadline exceeded
Oct 6 23:00:23.795148801 (321h30m11.598613394s) [seg_pos={0 0}, upload=282911]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 23:00:23.910977478 (321h30m11.714442086s) [upload=282911]: Done uploading segments...
Oct 6 23:00:23.911014610 (321h30m11.714479191s) [upload=282911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.1190224s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X........X.] duration:
2.776987062s
FAIL (2025-10-06 23:00:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.77702167s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.......X] duration:
1m0.000514338s
FAIL (2025-10-06 23:00:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00055159s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X........X] duration:
30.10961431s
FAIL (2025-10-06 23:00:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ing segments...
Oct 6 22:59:34.773117150 (321h29m22.576581742s) [upload=282913]: Got next segment. Inline: false
Oct 6 22:59:34.773168147 (321h29m22.576632739s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:59:34.773173665 (321h29m22.576638259s) [upload=282913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:59:34.883128853 (321h29m22.686593449s) [seg_pos={0 0}, upload=282913]: Begin upload segment...
Oct 6 22:59:34.883196370 (321h29m22.686660965s) [seg_pos={0 0}, upload=282913]: Done begin upload segment.
Oct 6 22:59:34.883221355 (321h29m22.686685953s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884198009 (321h29m22.687662601s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884476247 (321h29m22.687940838s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.884797902 (321h29m22.688262496s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885129708 (321h29m22.6885943s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885392103 (321h29m22.688856694s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.885650214 (321h29m22.689114805s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.894384549 (321h29m22.697849143s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897131662 (321h29m22.700596241s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.897440495 (321h29m22.700905093s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.898940881 (321h29m22.702405482s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.899674583 (321h29m22.703139176s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.900689546 (321h29m22.70415414s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902009925 (321h29m22.705474516s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.902390241 (321h29m22.705854818s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:34.903639900 (321h29m22.70710449s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Uploading piece...
Oct 6 22:59:35.515159175 (321h29m23.318623786s) [upload=282913]: Next returned nil segment
Oct 6 22:59:35.515197634 (321h29m23.318662231s) [upload=282913]: Waiting for error group managing segments...
Oct 6 22:59:35.651802100 (321h29m23.455266678s) [noise=true , address=145.79.227.2:4114 , piece_id=2B6KEDV36JAOBNB3OVMPYFSZGHM7CIDSTXV64AQJ3FYNCMJYKCSQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.663909364 (321h29m23.467373949s) [noise=true , address=145.79.227.12:4767 , piece_id=7AFEAD2RRKA6CMNE4STKULMKI775UHX7B3RVDLGPHTTMPHIMU44Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.664794249 (321h29m23.468258831s) [noise=true , address=145.79.227.7:4293 , piece_id=OBBN3XOPTNMLL3GTHE63P67XVGFGLU3O232WOFIYA74KZUOV5UIA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.669222290 (321h29m23.472686882s) [noise=true , address=145.79.227.14:4840 , piece_id=7B3SQAIC5AFCBGYKPID63UODPSNNR2UMXVXI55ACTXDQCPB6VVCA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675774948 (321h29m23.479239526s) [noise=true , address=145.79.227.17:4969 , piece_id=NYE6EVGNQL2EDUBXOQLT7YEVW6VATUMUGNQ4ZVWMQOEYRYOUOZVQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.675983428 (321h29m23.479448021s) [noise=true , address=145.79.227.4:4187 , piece_id=VCHJZM7EMODSHC5VTLVOK4KYRFXLBDBTEDAWOYLVZAUG552TTW4Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.680587049 (321h29m23.484051641s) [noise=true , address=145.79.227.16:4909 , piece_id=E76U6SA3UQG3T4EMZGKN4PLNNC32B33DZER5JDBFZPMLWAAAM5VA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 22:59:35.683200435 (321h29m23.486665013s) [noise=true , address=145.79.227.15:4874 , piece_id=NDJGXIG4NZXF7FAW45UBDT5BTADIKHTNP6NUXZPQ2O6FPB7WF2ZQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: <nil>
Oct 6 23:00:04.482633190 (321h29m52.286097784s) [noise=true , address=77.90.31.11:4554 , piece_id=ISY4SKEFVMSRMRQ3VXABTLAJ7F3BYHO2EHFMLVDURY2NA55S6WCQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.482919071 (321h29m52.286383665s) [noise=true , address=45.91.116.13:4153 , piece_id=II65PJRZHS3P4EJ3UYC5J5S3MJ5U3VDOABFBYAWDRFNO77AYPXRQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483083379 (321h29m52.286547977s) [noise=true , address=77.90.31.10:4519 , piece_id=2E6QTEFVK2TM7OVN32BJM4L5JSK3VB2IVXAMFLJDJPQYSTIO4QWA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 139.59.144.22:50470->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483213332 (321h29m52.286677931s) [noise=true , address=45.91.116.13:4155 , piece_id=JXRYZDG2FXO5NHRJVDSRL2GX25ATQWSJPIAVVDUEQKJYDRCOIA7Q, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:55178->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483398994 (321h29m52.286863591s) [noise=true , address=77.90.31.15:4699 , piece_id=IU3DTH272HY3UPHTD6QB4VDOYB4O6FNXMZCBSHF6DG2LQVBRFQHA, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:33128->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483526497 (321h29m52.286991081s) [noise=true , address=45.91.116.8:4048 , piece_id=6SDCJYMOSLCT7MZL7T5AEOJJ4PTOHTAY3EV7IYJQXHOTXSOZHELQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 139.59.144.22:46380->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483650237 (321h29m52.287114832s) [noise=true , address=45.91.116.12:4227 , piece_id=O3IJRTDHST4VE42CALXYJPEUOKIJ3PJAISXOIU7BS4HZIES77POQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:53980->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483806216 (321h29m52.287270812s) [noise=true , address=45.91.116.14:4190 , piece_id=AUQDG2BBCJJ2M6RNPEGJ2DWOGPIN7ZONCU64HRQPSFB6357JMOKQ, seg_pos={0 0}, upload=282913]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 139.59.144.22:41606->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:00:04.483836757 (321h29m52.287301337s) [seg_pos={0 0}, upload=282913]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:00:04.589615554 (321h29m52.393080166s) [upload=282913]: Done uploading segments...
Oct 6 23:00:04.589650592 (321h29m52.393115189s) [upload=282913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.109645486s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.807243921s
FAIL (2025-10-06 22:59:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.807268334s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.422048445s
FAIL (2025-10-06 22:59:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.422084266s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.66892921s
FAIL (2025-10-06 22:58:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.668941986s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.671691355s
FAIL (2025-10-06 22:58:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.671744008s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
3.614840175s
FAIL (2025-10-06 22:58:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.614918508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.488727168s
FAIL (2025-10-06 22:58:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.48876073s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.88799336s
FAIL (2025-10-06 22:57:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.888022918s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.763723032s
FAIL (2025-10-06 22:57:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.763755051s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.701303642s
FAIL (2025-10-06 22:57:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.701335963s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.5921631s
FAIL (2025-10-06 22:57:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.592198275s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.814621606s
FAIL (2025-10-06 22:56:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.814678125s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.790435526s
FAIL (2025-10-06 22:56:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.790449362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.551718465s
FAIL (2025-10-06 22:56:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.551752589s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.580105327s
FAIL (2025-10-06 22:56:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.580137825s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.703825927s
FAIL (2025-10-06 22:56:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.703876997s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
3.073073759s
FAIL (2025-10-06 22:56:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.073115125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.51426719s
FAIL (2025-10-06 22:56:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.514296676s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.682917826s
FAIL (2025-10-06 22:55:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.682951552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.656294293s
FAIL (2025-10-06 22:55:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.656392693s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.770851855s
FAIL (2025-10-06 22:55:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.770874136s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
3.202577331s
FAIL (2025-10-06 22:55:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.202604307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.837486189s
FAIL (2025-10-06 22:54:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.837518742s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.792792781s
FAIL (2025-10-06 22:54:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.792823917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.782448252s
FAIL (2025-10-06 22:54:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.782480926s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.568777667s
FAIL (2025-10-06 22:54:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.568809767s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
3.006492953s
FAIL (2025-10-06 22:54:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.006521247s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.640894955s
FAIL (2025-10-06 22:54:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.641163606s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.563849004s
FAIL (2025-10-06 22:53:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.56395894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.73187264s
FAIL (2025-10-06 22:53:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.731901429s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
12.089149054s
FAIL (2025-10-06 22:53:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.089163118s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.000856391s
FAIL (2025-10-06 22:53:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000884078s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.760283862s
FAIL (2025-10-06 22:53:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.760404685s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
25.178060321s
FAIL (2025-10-06 22:52:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 25.178085522s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.516365444s
FAIL (2025-10-06 22:52:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.516384973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.110863895s
FAIL (2025-10-06 22:52:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837450923 (321h21m43.640915517s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.837746517 (321h21m43.641211108s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.838078923 (321h21m43.641543517s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839295036 (321h21m43.642759636s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.839737614 (321h21m43.643202212s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.841802119 (321h21m43.645266713s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843215985 (321h21m43.646680582s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.843918826 (321h21m43.647383423s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.847034063 (321h21m43.650498653s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.849936736 (321h21m43.653401326s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:55.852057350 (321h21m43.655521951s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Uploading piece...
Oct 6 22:51:56.437181449 (321h21m44.240646041s) [upload=282798]: Next returned nil segment
Oct 6 22:51:56.437194503 (321h21m44.240659095s) [upload=282798]: Waiting for error group managing segments...
Oct 6 22:51:56.621774513 (321h21m44.425239091s) [noise=true , address=145.79.227.44:4225 , piece_id=VMXBMMZYIF77KFOLGSUAQWXPLACW2MK32DA7VWWXIYZ6J5RG4GCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.621931773 (321h21m44.425396351s) [noise=true , address=145.79.227.22:4519 , piece_id=YZRNLOAN5YBX734URL2EB2YCLXYVXATCHZKLGWCXOXX2QU63XRUQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.632506254 (321h21m44.435970833s) [noise=true , address=145.79.227.31:4802 , piece_id=YPNDUTRTU7IYGARW2SXIUBEG5SOGWEWRZ4HC4YHD7BQPEC7JZ4UQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.638511638 (321h21m44.441976227s) [noise=true , address=145.79.227.43:4171 , piece_id=BTVHWDDV4636C3KSKHJRQI3L7RSO5K46S7EBSOWIZJWOIGFKPY6Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:56.645212964 (321h21m44.448677548s) [noise=true , address=145.79.227.30:4792 , piece_id=D3344KOSAB34R26FAEZYQMTHCCNOFJKQOTV46PGTDJEAKCZDGFIA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:51:57.099414812 (321h21m44.902879399s) [noise=true , address=64.49.13.155:4914 , piece_id=YB4BAA33YQTC4E6ZOGASTE7MDR46ZUJVNXK2VEMTPCZKIILSNNQA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: <nil>
Oct 6 22:52:55.433094081 (321h22m43.236558673s) [noise=true , address=45.91.116.13:4162 , piece_id=RLDKWRPWM3WX6O42BTHBL75LSBSPRBEVDYEHXDBUQ4IN3X5P3OTQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: rpc: dial tcp 45.91.116.13:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433274540 (321h22m43.236739125s) [noise=true , address=77.90.31.16:4763 , piece_id=G4RVFEBK4O4QTYBB5AMTL2V6UIF5SCWSCIASBTRDWXUJPINBAK6A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: rpc: dial tcp 77.90.31.16:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433441208 (321h22m43.236905798s) [noise=true , address=45.91.116.5:4316 , piece_id=4V7EM5VI4OHF7CXLPK6EXHP2QUVYT42PK3BFT6243VZ42OBEYTKA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: read tcp 139.59.144.22:44382->45.91.116.5:4316: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433509768 (321h22m43.23697436s) [noise=true , address=77.90.31.18:4812 , piece_id=BZVDTBRIDDP4PYS553QMYDGGC442A5B4PGK7LBMJQ4RGMFIWQS5Q, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: read tcp 139.59.144.22:49462->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433558250 (321h22m43.237022839s) [noise=true , address=45.91.116.6:4986 , piece_id=M2HAEBACWUTJYNE5PVV7EZVH5KVERGUFCAEU5FODBHLSHOY5E2EA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: read tcp 139.59.144.22:45000->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433670048 (321h22m43.237134631s) [noise=true , address=77.90.31.12:4617 , piece_id=D2P2E5AH3QQRZD2MKT2UOUG7Y4QOARXSFU7MPBVQF2VFCMMYERCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:47744->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433755331 (321h22m43.237219914s) [noise=true , address=45.91.116.8:4052 , piece_id=JJ472TZ53QOWOCOC45SNCEWQE76NJO4LM4NPB6TFK2FQWVKACB4A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: read tcp 139.59.144.22:33770->45.91.116.8:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433781714 (321h22m43.237246302s) [noise=true , address=45.91.116.4:4962 , piece_id=QPJBUB5IFOCI2WC2QHHNHVGE47U5BNTE2Y7CUKM7EDLOGAAAG4RA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 139.59.144.22:40746->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.433897621 (321h22m43.237362213s) [noise=true , address=45.91.116.11:4147 , piece_id=6U24K2NHSU2KJ4AH3YSM7ZLKQNKH55ASKFHN42W3PCZYWJS6VREA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, address:45.91.116.11:4147): protocol: read tcp 139.59.144.22:49742->45.91.116.11:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434092748 (321h22m43.237557339s) [noise=true , address=77.90.31.17:4777 , piece_id=WCHVJ5MDVIGSQWE4VYSG2BRCIYFKGWBAZCQGDDVEEIO4ZIK4MGCQ, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: read tcp 139.59.144.22:41604->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434160736 (321h22m43.237625314s) [noise=true , address=45.91.116.9:4284 , piece_id=TTL5JXNQ7KOIFBQU7HQOGWCDNSWG5ZFIOXEZI22WLZ326H7YO25A, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:127CKUmRzR3215HNASZzQmbiy6shYsv98XjmQ2uGu4NjA37KKXx, address:45.91.116.9:4284): protocol: read tcp 139.59.144.22:55276->45.91.116.9:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434178177 (321h22m43.237642775s) [noise=true , address=45.91.116.7:4028 , piece_id=4DJWSDSQ45GSWCZH6H6OVGFWLAXRH27QI4CQAUKJG7Y45KDOV7SA, seg_pos={0 0}, upload=282798]: Done uploading piece. err: ecclient: upload failed (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB, address:45.91.116.7:4028): protocol: write tcp 139.59.144.22:41728->45.91.116.7:4028: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:55.434211421 (321h22m43.237676012s) [seg_pos={0 0}, upload=282798]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 6 22:52:55.542478957 (321h22m43.345943538s) [upload=282798]: Done uploading segments...
Oct 6 22:52:55.542508398 (321h22m43.34597299s) [upload=282798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.110890534s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001936791s
FAIL (2025-10-06 22:52:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001991246s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.113609321s
FAIL (2025-10-06 22:52:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
3.266647152 (321h21m51.070111746s) [upload=282801]: Starting upload
Oct 6 22:52:03.266773242 (321h21m51.070237837s) [upload=282801]: Done starting upload
Oct 6 22:52:03.266937035 (321h21m51.070401633s) [upload=282801]: Uploading segments...
Oct 6 22:52:03.267176514 (321h21m51.070641227s) [upload=282801]: Got next segment. Inline: false
Oct 6 22:52:03.268089244 (321h21m51.071553852s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:52:03.268111522 (321h21m51.071576123s) [upload=282801]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:52:03.430037228 (321h21m51.233501827s) [seg_pos={0 0}, upload=282801]: Begin upload segment...
Oct 6 22:52:03.430113862 (321h21m51.233578456s) [seg_pos={0 0}, upload=282801]: Done begin upload segment.
Oct 6 22:52:03.430191012 (321h21m51.233655604s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430232477 (321h21m51.233697077s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.430824279 (321h21m51.234288885s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.434338118 (321h21m51.23780271s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435142913 (321h21m51.238607514s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.435954054 (321h21m51.239418669s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.436658415 (321h21m51.240123013s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437196793 (321h21m51.240661376s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.437784701 (321h21m51.241249295s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.438473814 (321h21m51.24193841s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.446137710 (321h21m51.249602322s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.448705223 (321h21m51.252169836s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.453854771 (321h21m51.257319379s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.462270326 (321h21m51.265734926s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.464373362 (321h21m51.267837968s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:03.465236516 (321h21m51.26870112s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Uploading piece...
Oct 6 22:52:04.132660648 (321h21m51.936125282s) [upload=282801]: Next returned nil segment
Oct 6 22:52:04.132711626 (321h21m51.936176233s) [upload=282801]: Waiting for error group managing segments...
Oct 6 22:52:04.268577053 (321h21m52.072041634s) [noise=true , address=145.79.227.14:4839 , piece_id=SVISSTD52AGG7E7JKNW5HV4ZAIHMQ3CD5PKAGY2O4AYK4PA2PWDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269076811 (321h21m52.072541395s) [noise=true , address=145.79.227.3:4151 , piece_id=3FOVZXVEB3253MLNM4XXQXB2MVTOKK3XJZR6H55SDJ4IEUBLLHDQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.269136691 (321h21m52.072601272s) [noise=true , address=145.79.227.12:4767 , piece_id=6IPOWLQUCRC2EZOV24BTJV3UQ2W47MW7YFCIK4W3VAYVVFI44MFQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.276452785 (321h21m52.079917379s) [noise=true , address=145.79.227.5:4222 , piece_id=QSJ4JXWZ3BR5H7BXWWJTDZP2BL667L67VEPD3HJMCXAULRYLL6TA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.279150664 (321h21m52.082615249s) [noise=true , address=145.79.227.17:4969 , piece_id=PAOB6PTNPO7SJP327WRVCJ2STRQQMIJEVLSCGWCAFRXRLRZAV44A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.282533290 (321h21m52.085997871s) [noise=true , address=145.79.227.5:4223 , piece_id=IZPHPGP4JPK6CYPXZB3O7AEZDR4CHGOSWHK25N45WVRXAWLH4IGQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.286605407 (321h21m52.090069991s) [noise=true , address=145.79.227.16:4910 , piece_id=HOCU23QR3XDR7Z5CXAY7J6E3D7DI4XMG5TDJCPVLPXIYZFIFFFQA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.291130118 (321h21m52.094594711s) [noise=true , address=145.79.227.18:4006 , piece_id=46DN34BCNPULTMOKSC2ECHPAZIB53LXIPLM2DWOKL56WJEWCEL4A, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:04.296637658 (321h21m52.100102253s) [noise=true , address=145.79.227.3:4150 , piece_id=4LD34HU6CUUC67PRQ2ZA7SC2DCAVMKKAXZECKTEB6U4D7VGJOYHA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: <nil>
Oct 6 22:52:32.905984135 (321h22m20.709448735s) [noise=true , address=45.91.116.9:4262 , piece_id=W4OXKGKAE35KCKDSNQL3S75QZPIONV5GLOVACWIC3M37ME5RBAAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 139.59.144.22:48884->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906132393 (321h22m20.709596977s) [noise=true , address=77.90.31.17:4770 , piece_id=GFMQGGGYDEUAJNZSMFJHTP5PXEC537E5AYUCRSMNQ674WVZY5TGA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906293048 (321h22m20.70975764s) [noise=true , address=77.90.31.10:4518 , piece_id=3MSB7KIKPCAFCHUI3UOCYFAEYG7W4I2A4I7W7MAOX6YXHVCKURUA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 139.59.144.22:59290->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906302895 (321h22m20.709767512s) [noise=true , address=77.90.31.11:4553 , piece_id=MGSQPY7RJXHKO7UVWEHGF7ND26PWKEXSSEGXDTH2C3ESIKD436MA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:47872->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906403245 (321h22m20.709867837s) [noise=true , address=45.91.116.11:4119 , piece_id=2AYCBJSFSSYYRGFEMR5UWT3OLIEEWXEZVTIUU4N6OUWRSVPIM7YQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 139.59.144.22:49046->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906429852 (321h22m20.709894447s) [noise=true , address=77.90.31.19:4841 , piece_id=ALRINMNMX7CYVWPIVSVRDPLHIHGZ2VUIGZLRGI5HTRPA5IEDMEAQ, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 139.59.144.22:37584->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906437501 (321h22m20.709902093s) [noise=true , address=45.91.116.12:4227 , piece_id=FLFWRGBERDRKWNZ4V4FXYCZOIPQULHGE3NYOSR2KL3TDPPC6Z4NA, seg_pos={0 0}, upload=282801]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:37362->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:52:32.906459259 (321h22m20.709923851s) [seg_pos={0 0}, upload=282801]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 22:52:33.017661298 (321h22m20.821125895s) [upload=282801]: Done uploading segments...
Oct 6 22:52:33.017697522 (321h22m20.821162119s) [upload=282801]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.113638369s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.680351435s
FAIL (2025-10-06 22:49:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.680378507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
3.377880802s
FAIL (2025-10-06 22:48:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.377919629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.822191157s
FAIL (2025-10-06 22:48:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.82222591s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.992311566s
FAIL (2025-10-06 22:47:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 22:04:40.124443389 (320h34m27.927907966s) [upload=282109]: Starting upload
Oct 6 22:04:40.124539230 (320h34m27.928003824s) [upload=282109]: Done starting upload
Oct 6 22:04:40.124671777 (320h34m27.928136367s) [upload=282109]: Uploading segments...
Oct 6 22:04:40.124928300 (320h34m27.928392877s) [upload=282109]: Got next segment. Inline: false
Oct 6 22:04:40.125013547 (320h34m27.928478124s) [upload=282109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 22:04:40.125018228 (320h34m27.928482805s) [upload=282109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 22:04:40.240598039 (320h34m28.044062631s) [seg_pos={0 0}, upload=282109]: Begin upload segment...
Oct 6 22:04:40.240655734 (320h34m28.044120316s) [seg_pos={0 0}, upload=282109]: Done begin upload segment.
Oct 6 22:04:40.240700158 (320h34m28.044164749s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.240706532 (320h34m28.044171132s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.241368237 (320h34m28.044832832s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.241657518 (320h34m28.045122113s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.242196456 (320h34m28.045661058s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.242470596 (320h34m28.045935177s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.242870747 (320h34m28.046335346s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.243396833 (320h34m28.046861427s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.243572746 (320h34m28.04703734s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.243925943 (320h34m28.047390536s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.244076216 (320h34m28.047540797s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.244531195 (320h34m28.047995789s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.244754757 (320h34m28.04821936s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.245254511 (320h34m28.048719092s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.245936892 (320h34m28.049401494s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.246772240 (320h34m28.050236836s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Uploading piece...
Oct 6 22:04:40.897188204 (320h34m28.700652796s) [upload=282109]: Next returned nil segment
Oct 6 22:04:40.897219546 (320h34m28.700684136s) [upload=282109]: Waiting for error group managing segments...
Oct 6 22:04:41.048159492 (320h34m28.851624076s) [noise=true , address=145.79.227.7:4295 , piece_id=UAA3LG42BPSKNPG4AU77YOYFFXOFKULVLAJ6XZUJKE2S23Z657XA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.049015249 (320h34m28.85247983s) [noise=true , address=145.79.227.14:4837 , piece_id=5IDBA7YT72YGMORIXN6MTHFDEGIGAZQKJPK7BVTJXUIXZRPYM5ZQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.050414372 (320h34m28.853878955s) [noise=true , address=145.79.227.17:4970 , piece_id=AXBKH6PC262KSHI4JMDT6YY5K5TTJAIPGXFFIQANP2WBSGOCQFNQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.051320052 (320h34m28.854784631s) [noise=true , address=145.79.227.15:4874 , piece_id=BQDGUCN5LG4Z7EOZMWJ7ZN4ETT66TDFDZMLKLSHUWT6XKU2AG3LA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.051571274 (320h34m28.855035852s) [noise=true , address=145.79.227.5:4223 , piece_id=W7IM5CKGIZUCEL4J5AFCJXDU3F7MQOW2OQ235GGABXC6WNA6U32A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.053376635 (320h34m28.856841217s) [noise=true , address=145.79.227.13:4804 , piece_id=63ZS7UUHAGOFLXVWSNHKMYZW3XS7MBOA3RWX2PNC3XGXF66SQOWA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.058058491 (320h34m28.861523083s) [noise=true , address=145.79.227.4:4185 , piece_id=NINUN52EB6SXBRVQ7H3IDEDLRWULNA5JVT3PSHUPT3QP4UDT3AEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.061265712 (320h34m28.864730303s) [noise=true , address=145.79.227.14:4838 , piece_id=VYJEY3NIZ4A7OHWFBQ32V3I5FCDQPTN5BKFKPJY5YODCO6VR2LKQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.068942363 (320h34m28.872406944s) [noise=true , address=145.79.227.7:4293 , piece_id=T2RJRAYDI6MSKYPYX4GFGYPHGT6NWKAOKTMKM5VQZ754QQRG246A, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:04:41.069059755 (320h34m28.872524339s) [noise=true , address=145.79.227.11:4729 , piece_id=DMJI6AY7QTHFQNACHSNILPSPF6VHU2J3FYLYXKT2T3WQDJX2G2UA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:05:06.625853604 (320h34m54.429318188s) [noise=true , address=45.91.116.13:4153 , piece_id=CQOSM6U7D2QHNEJXAXTWAKGBU2TA2FXJRWMWOTWKS5C7IWOSYUPA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:05:06.881003880 (320h34m54.684468458s) [noise=true , address=77.90.31.17:4770 , piece_id=3PZ26VTEFYO4QPS55GCWQVIDHJ335IYUCZ2LMAZZJ6WS63HYECUA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: <nil>
Oct 6 22:05:09.828604831 (320h34m57.632069414s) [noise=true , address=45.91.116.5:4299 , piece_id=K7B6HEYCWL727OQKWLI74BDPSUSYF3J7AO343F2JAEI33KHMBT3Q, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 139.59.144.22:43970->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:05:09.828746252 (320h34m57.632210833s) [noise=true , address=45.91.116.9:4263 , piece_id=YSEBB5FS2452GPIZMEYB2SOVJSPCN4VCP7VTPD6TYMFCLHUSSQEQ, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 139.59.144.22:33152->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:05:09.828787351 (320h34m57.632251935s) [noise=true , address=77.90.31.16:4733 , piece_id=EAURYAED4RXZLSPBWK2AKTIF7ZZE5XMQVSNDO4MWE3QTH3KXDRHA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:05:09.828829118 (320h34m57.632293717s) [noise=true , address=77.90.31.12:4590 , piece_id=N7ALXLS33FQB5LU23F5D4LB4LLRAI32NIETJ2RJMOHOCMSKFM4CA, seg_pos={0 0}, upload=282109]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:41962->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 22:05:09.828883864 (320h34m57.632348456s) [seg_pos={0 0}, upload=282109]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 22:05:09.936174235 (320h34m57.73963882s) [upload=282109]: Done uploading segments...
Oct 6 22:05:09.936254043 (320h34m57.739718636s) [upload=282109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.992347608s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
3.409484135s
FAIL (2025-10-06 22:47:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.40951286s
core
uplink
us-select-1
us1-placement-20
|