VMware vCenter Server 7.0.3.00100 - HTTP Status 500 – Internal Server Error

Jelenség: VCSA webes felületen nem lehet belépni, a menedzsment felület egy "HTTP Status 500 – Internal Server Error" hibával fogad. Egyéb hibajelenség nincs, minden service fut.

Hiba: SSH belépés után kiderül, hogy a MACHINE_SSL_CERT lejárt.

Megoldás: 

root@vcenter7 [ ~ ]# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
[*] Store : MACHINE_SSL_CERT
Alias :    __MACHINE_CERT
            Not After : Nov  9 00:28:15 2023 GMT
[*] Store : TRUSTED_ROOTS
Alias :    b31084f54b1a7cbf10e6f3bf5e2f784be75183bc
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : machine
Alias :    machine
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vsphere-webclient
Alias :    vsphere-webclient
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vpxd
Alias :    vpxd
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vpxd-extension
Alias :    vpxd-extension
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : hvc
Alias :    hvc
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : data-encipherment
Alias :    data-encipherment
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : APPLMGMT_PASSWORD
[*] Store : SMS
Alias :    sms_self_signed
            Not After : Nov  8 12:33:01 2031 GMT
[*] Store : wcp
Alias :    wcp
            Not After : Nov  3 12:28:14 2031 GMT

Megújítani a webes felületen lehet, de abba nem lehet belépni (Internal Server Error).

Akkor újítsuk meg parancssorból:

Note: The Name, Hostname and VMCA values should match the PNID of the Node where you are replacing the Certificates. PNID should always match the Hostname.

Ehhez először szerezzük meg a szerver PNID-jét, ha nem tudjuk fejből:

root@vcenter7 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
foo.bar.hu

root@vcenter7 [ ~ ]# hostname -f
foo.bar.hu

Ha megvan, futtassuk a Certificate Manager-t, majd válasszuk a 3-as pontot, hiszen a Machine SSL certificate járt le:

root@vcenter7 [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
		 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
		|                                                                     |
		|      *** Welcome to the vSphere 6.8 Certificate Manager  ***        |
		|                                                                     |
		|                   -- Select Operation --                            |
		|                                                                     |
		|      1. Replace Machine SSL certificate with Custom Certificate     |
		|                                                                     |
		|      2. Replace VMCA Root certificate with Custom Signing           |
		|         Certificate and replace all Certificates                    |
		|                                                                     |
		|      3. Replace Machine SSL certificate with VMCA Certificate       |
		|                                                                     |
		|      4. Regenerate a new VMCA Root Certificate and                  |
		|         replace all certificates                                    |
		|                                                                     |
		|      5. Replace Solution user certificates with                     |
		|         Custom Certificate                                          |
		|         NOTE: Solution user certs will be deprecated in a future    |
		|         release of vCenter. Refer to release notes for more details.|
		|                                                                     |
		|      6. Replace Solution user certificates with VMCA certificates   |
		|                                                                     |
		|      7. Revert last performed operation by re-publishing old        |
		|         certificates                                                |
		|                                                                     |
		|      8. Reset all Certificates                                      |
		|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 3

Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@foobar.local
Enter password:

Miután megadtuk a megfelelő credenciálisokat, olyan jön, ami a leírásban nem szerepel:

Certificate Manager tool do not support vCenter HA systems

Hmm, WTF HA systems? Nincs itt semmiféle HA systems, te nyomorult .... Keresés után kiderül, hogy csak a VMware mérnökei voltak lámák, de egy workraound segíthet:


When I got the "Certificate Manager tool do not support vCenter HA systems" error the following solution worked for me:

1. mkdir /var/tmp/vmware

2. Run certificate-manager again

I hope it helps.

#worksformetoo, haladjunk:

root@vcenter7 [ ~ ]#  mkdir /var/tmp/vmware
root@vcenter7 [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
		 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
		|                                                                     |
		|      *** Welcome to the vSphere 6.8 Certificate Manager  ***        |
		|                                                                     |
		|                   -- Select Operation --                            |
		|                                                                     |
		|      1. Replace Machine SSL certificate with Custom Certificate     |
		|                                                                     |
		|      2. Replace VMCA Root certificate with Custom Signing           |
		|         Certificate and replace all Certificates                    |
		|                                                                     |
		|      3. Replace Machine SSL certificate with VMCA Certificate       |
		|                                                                     |
		|      4. Regenerate a new VMCA Root Certificate and                  |
		|         replace all certificates                                    |
		|                                                                     |
		|      5. Replace Solution user certificates with                     |
		|         Custom Certificate                                          |
		|         NOTE: Solution user certs will be deprecated in a future    |
		|         release of vCenter. Refer to release notes for more details.|
		|                                                                     |
		|      6. Replace Solution user certificates with VMCA certificates   |
		|                                                                     |
		|      7. Revert last performed operation by re-publishing old        |
		|         certificates                                                |
		|                                                                     |
		|      8. Reset all Certificates                                      |
		|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 3

Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@foobar.local
Enter password:

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] : 

Enter proper value for 'Name' [Default value : CA] : 

Enter proper value for 'Organization' [Default value : VMware] : 

Enter proper value for 'OrgUnit' [Default value : VMware Engineering] : 

Enter proper value for 'State' [Default value : California] : 

Enter proper value for 'Locality' [Default value : Palo Alto] : 

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 

Enter proper value for 'Email' [Default value : email@acme.com] : 

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : foo.bar.hu

Enter proper value for VMCA 'Name' : foo.bar.hu

You are going to regenerate Machine SSL cert using VMCA
Continue operation : Option[Y/N] ? : y

Persze, hogy "y" csak haladjunk, mert várnak a HUP flame-ek ....

Get site nameompleted [Replacing Machine SSL Cert...]                  
default-first-site
Lookup all services
Get service default-first-site:32c3fac4-ec71-4553-93b5-e01d5cdbb68a
Update service default-first-site:32c3fac4-ec71-4553-93b5-e01d5cdbb68a; spec: /tmp/svcspec_f78rv0at
Get service default-first-site:6a88e16d-cd73-46d4-8c82-159359ce6d11
Update service default-first-site:6a88e16d-cd73-46d4-8c82-159359ce6d11; spec: /tmp/svcspec_z9zjad_d
Get service default-first-site:93596278-a162-49d1-b60b-141beb527092
Update service default-first-site:93596278-a162-49d1-b60b-141beb527092; spec: /tmp/svcspec_l3u17n0_
Get service 367d6898-8ea6-4c4b-b19e-6e68c0e791e0
Update service 367d6898-8ea6-4c4b-b19e-6e68c0e791e0; spec: /tmp/svcspec_8cba5n17
Get service 0061d7a4-31b1-47fe-859a-db9906bc200d
Update service 0061d7a4-31b1-47fe-859a-db9906bc200d; spec: /tmp/svcspec_5ie671pj
Get service f3ddbe0b-2062-4db6-9ca4-b3ae24215c71
Update service f3ddbe0b-2062-4db6-9ca4-b3ae24215c71; spec: /tmp/svcspec_gbkzk_07
Get service 9ed34e67-c9ab-432a-8434-6a165c7a4997
Update service 9ed34e67-c9ab-432a-8434-6a165c7a4997; spec: /tmp/svcspec_dz0v9wdz
Get service 80bfac56-f232-4395-8da2-369bd0003e93
Update service 80bfac56-f232-4395-8da2-369bd0003e93; spec: /tmp/svcspec_sgbthrxl
Get service 1425f06c-bda2-4f21-b1c6-384988f6eea6
Update service 1425f06c-bda2-4f21-b1c6-384988f6eea6; spec: /tmp/svcspec_injv9f1h
Get service 1b88ef8e-74d8-4998-865e-98312faa2fa6
Update service 1b88ef8e-74d8-4998-865e-98312faa2fa6; spec: /tmp/svcspec_revgf_zv
Get service b953c602-0b51-41b0-9742-ebf946853336
Update service b953c602-0b51-41b0-9742-ebf946853336; spec: /tmp/svcspec_uqoxzfd0
Get service 32f9b432-5701-4720-8a11-9e09b668a3e9_com.vmware.vsphere.client
Don't update service 32f9b432-5701-4720-8a11-9e09b668a3e9_com.vmware.vsphere.client
Get service e921c560-3903-4725-a815-03899c1cde61
Update service e921c560-3903-4725-a815-03899c1cde61; spec: /tmp/svcspec_td00jgxq
Get service 5afd3a63-dbe3-4608-a528-fe481007d9ad
Update service 5afd3a63-dbe3-4608-a528-fe481007d9ad; spec: /tmp/svcspec_n30zw1ip
Get service 58fe6fb1-2834-48b0-8390-703ead6a5648
Update service 58fe6fb1-2834-48b0-8390-703ead6a5648; spec: /tmp/svcspec_2lp_8qtj
Get service 58fe6fb1-2834-48b0-8390-703ead6a5648_kv
Update service 58fe6fb1-2834-48b0-8390-703ead6a5648_kv; spec: /tmp/svcspec_apuprpdb
Get service d00fb6cc-b6d3-4efb-8f22-8d6f0afa915a
Update service d00fb6cc-b6d3-4efb-8f22-8d6f0afa915a; spec: /tmp/svcspec_4lwn1um1
Get service 58fe6fb1-2834-48b0-8390-703ead6a5648_authz
Update service 58fe6fb1-2834-48b0-8390-703ead6a5648_authz; spec: /tmp/svcspec_r3d9btka
Get service ba8f687d-4c7d-4f68-a95b-8f51acb63f7d
Update service ba8f687d-4c7d-4f68-a95b-8f51acb63f7d; spec: /tmp/svcspec_zh4h4jnw
Get service 9447fa57-34c8-4ac8-be86-80fb19759ac8
Update service 9447fa57-34c8-4ac8-be86-80fb19759ac8; spec: /tmp/svcspec_zqkq_v5e
Get service b1de4928-5c2e-4b9a-b1e5-2702e9ee12c8
Update service b1de4928-5c2e-4b9a-b1e5-2702e9ee12c8; spec: /tmp/svcspec_uddzb9ch
Get service febaef36-b88b-46b8-8407-d92628aea88b
Update service febaef36-b88b-46b8-8407-d92628aea88b; spec: /tmp/svcspec_eoxennnk
Get service 32f9b432-5701-4720-8a11-9e09b668a3e9
Update service 32f9b432-5701-4720-8a11-9e09b668a3e9; spec: /tmp/svcspec_z9e5ny_z
Get service be2e7282-1b57-4e54-90d2-26bbdb031ad1
Update service be2e7282-1b57-4e54-90d2-26bbdb031ad1; spec: /tmp/svcspec_xklexbfw
Get service 1c25b2b2-2018-4354-a24a-ad802370a5be
Update service 1c25b2b2-2018-4354-a24a-ad802370a5be; spec: /tmp/svcspec_8lgf3ah8
Get service 27992d4c-a5ee-4239-a6a8-8c381c7b3785
Update service 27992d4c-a5ee-4239-a6a8-8c381c7b3785; spec: /tmp/svcspec_79aw3xuq
Get service 4cb5bac4-3d14-4364-8e42-88d699183e07
Update service 4cb5bac4-3d14-4364-8e42-88d699183e07; spec: /tmp/svcspec_hckegci9
Get service 6bff08f3-e97f-4ec1-b43f-832fd49d0ddb
Dont update service 6bff08f3-e97f-4ec1-b43f-832fd49d0ddb
Get service 33f7524f-8fa2-4ed9-a7a4-eeba7f807026
Update service 33f7524f-8fa2-4ed9-a7a4-eeba7f807026; spec: /tmp/svcspec_by9fha0n
Get service 0fabb220-4510-4aa8-a16b-28ad294df844
Update service 0fabb220-4510-4aa8-a16b-28ad294df844; spec: /tmp/svcspec_olejmag2
Get service dbd74828-847c-4621-bf95-d011230997b6
Update service dbd74828-847c-4621-bf95-d011230997b6; spec: /tmp/svcspec_sh8rpmfm
Get service ed157798-2a17-4553-aa12-d1dce524c629
Update service ed157798-2a17-4553-aa12-d1dce524c629; spec: /tmp/svcspec_01cojgy4
Get service fe6ee610-48e2-438f-a567-d81f7cbf9e62
Update service fe6ee610-48e2-438f-a567-d81f7cbf9e62; spec: /tmp/svcspec_dnbiae25
Get service 9119d476-e950-488a-8921-f390ffd8beaa
Update service 9119d476-e950-488a-8921-f390ffd8beaa; spec: /tmp/svcspec_exs7k0n_
Get service 88402717-c2ea-451d-9e0f-ec11554efc7b
Update service 88402717-c2ea-451d-9e0f-ec11554efc7b; spec: /tmp/svcspec_93phvtxo
Get service 18b626cc-326b-4e2b-a01e-fc3d86eba666
Update service 18b626cc-326b-4e2b-a01e-fc3d86eba666; spec: /tmp/svcspec_nz04lw5u
Get service ebdb0adf-acf2-4801-a7a3-52317ed93615
Update service ebdb0adf-acf2-4801-a7a3-52317ed93615; spec: /tmp/svcspec__02y2ez1
Get service 117ab9ae-0c60-4b24-b900-cb3ebb6d7d14
Update service 117ab9ae-0c60-4b24-b900-cb3ebb6d7d14; spec: /tmp/svcspec_s38uv6fo
Get service c0dcb8cc-f085-4674-8c08-d547be740d10
Update service c0dcb8cc-f085-4674-8c08-d547be740d10; spec: /tmp/svcspec_a8wqduae
Get service 1f43fd43-8dfe-4e64-8469-4f030ac1e4a0
Update service 1f43fd43-8dfe-4e64-8469-4f030ac1e4a0; spec: /tmp/svcspec_jcvf47cd
Get service 7b589d87-785d-4241-bd03-4637c20dfd86
Update service 7b589d87-785d-4241-bd03-4637c20dfd86; spec: /tmp/svcspec_ii4v1ex3
Get service c43154be-ef59-4902-86c4-5cd15f347c4b
Update service c43154be-ef59-4902-86c4-5cd15f347c4b; spec: /tmp/svcspec_eaiy33w1
Get service 46046137-90e0-49d6-a11d-25a322f8f919
Update service 46046137-90e0-49d6-a11d-25a322f8f919; spec: /tmp/svcspec_7x5ybo7a
Get service 6caa48cb-2757-4651-8be6-0f679dae9847
Update service 6caa48cb-2757-4651-8be6-0f679dae9847; spec: /tmp/svcspec_lkk7ello
Get service dabca688-6711-4142-8fa0-de8cc52d53c7
Update service dabca688-6711-4142-8fa0-de8cc52d53c7; spec: /tmp/svcspec_vtd4u0xm
Updated 43 service(s)
Status : 100% Completed [All tasks completed successfully]

Nézzük mi lett a tanúsítvánnyal:

root@vcenter7 [ ~ ]# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
[*] Store : MACHINE_SSL_CERT
Alias :	__MACHINE_CERT
            Not After : Nov 12 13:29:02 2025 GMT
[*] Store : TRUSTED_ROOTS
Alias :	b31084f54b1a7cbf10e6f3bf5e2f784be75183bc
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : machine
Alias :	machine
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vsphere-webclient
Alias :	vsphere-webclient
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vpxd
Alias :	vpxd
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : vpxd-extension
Alias :	vpxd-extension
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : hvc
Alias :	hvc
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : data-encipherment
Alias :	data-encipherment
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : APPLMGMT_PASSWORD
[*] Store : SMS
Alias :	sms_self_signed
            Not After : Nov  8 12:33:01 2031 GMT
[*] Store : wcp
Alias :	wcp
            Not After : Nov  3 12:28:14 2031 GMT
[*] Store : BACKUP_STORE
Alias :	bkp___MACHINE_CERT
            Not After : Nov  9 00:28:15 2023 GMT
Alias :	bkp_machine
            Not After : Nov  3 12:28:14 2031 GMT
Alias :	bkp_vsphere-webclient
            Not After : Nov  3 12:28:14 2031 GMT
Alias :	bkp_vpxd
            Not After : Nov  3 12:28:14 2031 GMT
Alias :	bkp_vpxd-extension
            Not After : Nov  3 12:28:14 2031 GMT
Alias :	bkp_hvc
            Not After : Nov  3 12:28:14 2031 GMT
Alias :	bkp_wcp
            Not After : Nov  3 12:28:14 2031 GMT

Megújult! Szuper!

Nocsak! Nézzük a webes klienst ... Működik!

Akkor vissza a flame-hez!

PS: mit jelent az, hogy "nameompleted" ? :D

PS2: lol @ "vSphere 6.8 Certificate Manager"

( NevemTeve | 2023. 11. 14., k – 07:23 )

Szerkesztve: 2023. 11. 14., k – 07:30

> Certificate Manager tool do not support vCenter HA systems

Ilyen megtévesztő hibaüzenetet úgy tudunk előállítani, ha a programunkban a hibakezelést kivételekkel (Execption, OnErrorGoto, Exlst, trap stb) oldjuk meg: egy-egy programrészt berakunk egy "try-block"-ba, és a blokk végén írunk hibakezelést ahhoz a hibához, ami szerintünk abban a blokkban a legvalószínűbb, illetve ami egyáltalán eszünkbe jut. Ha másfajta hiba lép fel, akkor vagy fals hibaüzenet megy ki, vagy a hiba "felfelé terjed" egy olyan hibakezelőhöz, ami még kevésbé tudja, hogy mi lehetett a a gond.

Most a konkrét esetben igen, mert egyszer valaki kidebuggolta (strace, pl.), hogy mi a valódi hiba. Ettől még a koncepcionális gond megmarad: ha korrektül akarunk hibakezelni, akkor minden nemtriviis műveletet (például fájl létrejozást) hibaellenőrizni kellene. Ezt Exception-okkal praktkiusan nem lehet megcsinálni, mert a kód teljesen olvashatatlanná válna. Ja meg lusták is vagyunk hozzá.