Monitoring Certificate Expiry Dates of a JAVA Keystore (.JKS) File in a WebLogic Domain (Shell Script, Scheduling, Alerting)
One of my client asked me how to monitor expiry dates of JKS certificate entries in a WebLogic domain. If an imported certificate entry expires in a JKS (Java Keystore File) then the HTTPS transactions is going to fail because of SSL handshake exception/failure.
So, a proactive administrator should monitor these certificates and should warn the 3rd party end point URL’s administrator or update/import the certificate with the new one before the failure. Especially, if you are working with OSB (Oracle Service Bus) and SOA Suite 11g/12c WebLogic domains it is very crucial to monitor imported JKS certificate entries.
Monitoring Certificate Expiry Dates of a JAVA Keystore (.JKS) File in a WebLogic Domain (Shell Script, Scheduling, Alerting)
How to monitor imported certificates (CRT, CER) expiry / expiration date in a JKS (Java Keystore) file and create remaining day alerts/notifications?
There are too many alternatives to do that. I have found some other scripts available but all of them based on JKS alias entries. Then decided to write a new one from scratch to iterate all JKS aliases in the JKS file.
I’ll explain two method for doing this JKS certificate monitoring in WebLogic.
- Shell scripting and OS Cron Job
- Shell script and WLSDM Scheduler
At the end of post I’ll put download link of the script and you can download it. Go to my GitHub project page to download entire project.
Step by Step Script Usage and Integration via Server CRON Job:
1. Download monitorJKS folder and upload to your server (Do it at the end, firstly read the whole tutorial)
2. Edit environment variables according to your server and WebLogic installation paths.
|
1 2 3 4 5 6 7 8 9 |
#!/bin/bash #Author: M.Fevzi Korkutata | Last day of 2017 (New year PARTY script) #Change environment variables: __scriptPath=/home/oracle/scripts/monitorJKS __keytool="/u01/weblogic/jdk8/jdk1.8.0_112/bin/keytool" __keystore="/u01/weblogic/Middleware12212/wlserver/server/lib/DemoTrust.jks" __keystorepass="DemoTrustKeyStorePassPhrase" __thresholdDay="7" |
3. Variable Info:
3.1. Change “__thresholdDay” value according to your preference. By default, I have set it with 7 which means it will start to send ALERT emails 7 days before the certificate expiration.
3.2. Do not forget about to set your local “__scriptPath” value.
4. Comment out below section for the SMTP email delivery. (Line 58)
|
1 2 |
#Comment out if you want to send as WARNING email. #cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo |
5. Create your CRON job for /path/to/monitorJKS/monitorJKSforWebLogic_Lnx.sh script. Running once in a day is just enough. Schedule for every midnight.
Step by Step Script Usage and Integration via WLSDM Scheduler:
Scheduling this script via WLSDM for WebLogic is more exciting and smarter. Because, I am going to use stdErr functionality of Shell and WLSDM interface.
1. Copy monitorJKSforWebLogic_Lnx.sh to server.
2. Go to “WLSDM Console > Configuration > Monitoring & Diagnostics > User Defined Actions / Scripts (tab)” page.
WLSDM Console Menu (Portlet in WebLogic Console)
3. In the page operations menu (on top-right of every page) click on “New User Defined Action / Script” button.
4. In the modal window add shell script file as below and save.
WLSDM: New User Defined Action / Script (Click to enlarge)
5. Switch to “Scheduled Jobs / Downtime” tab.
6. In the page operation menu (on top-right of every page) click on “New Scheduled Jobs / Downtime” button.
7. In the modal window add new CRON job as below and save. (you can choose every minute for testing the cron job temporarily)
WLSDM: New Scheduled Job / Downtime (Click to enlarge)
That’s it! The integration and monitoring of JKS certificates expiry date is done. Let’s test it and see the results.
I chose every minute to test the script and understand that WLSDM scheduler is working properly. The threshold in the script is 100 days which is a very high value; in a production system it must be something like 5 or 7 days. In my test case the threshold is:
__thresholdDay="100"
Let’s see the WLSDM scheduler output (enlarge below screen capture). Here is the job records and its results. It demonstrates all the JKS monitoring processes passed successfully and we will not get any email alert from WLSDM.
WLSDM for WebLogic: Scheduled Job Execution History (click to enlarge)
Drill down each monitoring cycle detail by clicking “View Output/Details” lens in the “Scheduled Job Execution History” list. WLSDM reflects all the “SHELL/BASH stdOut” and “echo/printf” message values in this modal window page. It is possible to see Linux/Unix/Windows shell/bash (terminal) scripts’ stdOut and stdErr values in this page. Click below screen capture to see a successful JKS monitoring output for my shell script/action which is monitorJKS.sh.
WLSDM CRON Job Detail Modal Window: Monitoring JKS certificate expiry date (click to enlarge)
As you see the remaining days for the alias volthreadcom is 124 days. The threshold is 100 which means remaining days are more than threshold; so the result is as expected: –> Script executed successfully! Certificates are OK.
For the second scenario; I am going to change the threshold from 100 to 150. In this case volthreadcom JKS certificate entry would be in trouble 
I am expecting to get a failed job result. Because the shell script will terminate itself with error (check “exit 1″ in the script)
After updating threshold to 150; the WLSDM cron job switched to failed status immediately. Check below screen captures!
WLSDM for WebLogic: Failed Scheduler CRON Jobs List (click to enlarge)
Click “View Details/Output” button to see what’s going on for the JKS certificate entries.
WLSDM Cron Job Details: View Warning for Certificate Expiry Dates (click to enlarge)
YESSS! Here is the most important part. I have received an email for the certificates that’s going to expire according to my threshold.
Click below image to enlarge and check the details for understanding how to receive HTML emails for the certificates going to expire.
JKS Certificate Expiry Date Warning: WLSDM for WebLogic Scheduler HTML Email Notification (click to enlarge)
Actually I am cheating WLSDM. Because; I know that WLSDM informs administrators about failed cron jobs/scripts by sending HTML email notifications as above. So, when I get an email about current “WLSDM WebLogic scheduled job” probably it means there is a WARNING about JKS certificate expiry date. Check below section in the script especially the (exit 1) part then you will understand what I mean. Otherwise please let me know to think about your case together
|
1 2 3 4 5 6 7 |
<strong>#__lcCEW = line count for certificate expire warning (file)</strong> if [ $__lcCEW -gt 0 ]; then (>&2 echo "!!! [WARNING] Check expired certificates !!!") (>&2 echo "$(cat $__scriptPath/certificateExpireWarning.txt)") #Comment out if you wanto send as WARNING email. #cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo exit 1 |
I have finished the documentation part and reading part. I have also prepared a Youtube tutorial/screencast about monitoring JKS entries. I’ve recorded the entire video on my iMac. The date function/command is different in MacOS (Sierra) than Linux. I have did some tricks for executing my shell script properly for macOS environment (If you are interested knock my door).
If you already read this technical blog post; please also watch the entire technical screencast it will inspire you about your PRODUCTION environment monitoring cases. Especially about WebLogic cron job integrations and definitions.
Youtube Screencast Tutorial: Monitoring Certificate Expiry Date of JKS
(Please share the screencast with other Oracle FMW WebLogic Community. Regards!) –> Sharing is just caring Fevzi Korkutata
Source Code (Script and sample output file):
Github URL: https://github.com/admineer/monitorJKSforWebLogic
All the project source and sample output files are available in the Github monitorJKSforWebLogic project. I am also adding the entire script as below. (Copy/Download the script from Github; because the shell script source file is well formated at there)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
#!/bin/bash #Author: M.Fevzi Korkutata | Last day of 2017 (New year PARTY script) #Change environment variables: __scriptPath=/home/oracle/scripts/monitorJKS __keytool="/u01/weblogic/jdk8/jdk1.8.0_112/bin/keytool" __keystore="/u01/weblogic/Middleware12212/wlserver/server/lib/DemoTrust.jks" __keystorepass="DemoTrustKeyStorePassPhrase" __thresholdDay="100" #Static Variables __currentDate=$(date +%s) __threshold=$(($__currentDate + ($__thresholdDay*24*60*60))) #Flush output values echo -n > $__scriptPath/certificateStatus.txt echo -n > $__scriptPath/certificateExpireWarning.txt echo -n > $__scriptPath/certificateSummary.txt #Fetch certificate "until" dates for i in $($__keytool -list -v -keystore $__keystore -storepass $__keystorepass | grep 'Alias name:' | perl -ne 'if(/name: (.*?)\n/) { print "$1\n"; }') do echo "$i valid until: "$($__keytool -list -v -keystore $__keystore -storepass $__keystorepass -alias "$i" | grep 'Valid from' | head -1 | perl -ne 'if(/until: (.*?)\n/) { print "$1\n"; }') >> $__scriptPath/certificateStatus.txt done #Calculate certificate remaining days __lc=$(cat $__scriptPath/certificateStatus.txt | wc -l) for (( c=1 ; c<=$__lc ; c++ )) do __alias=$(awk "NR==$c" $__scriptPath/certificateStatus.txt | awk '{print $1}') __until=$(awk "NR==$c" $__scriptPath/certificateStatus.txt | perl -ne 'if(/until: (.*?)\n/) { print "$1\n"; }') #echo $__until __untilSeconds=`date -d "$__until" +%s` __remainingDays=$(( ($__untilSeconds - $(date +%s)) / 60 / 60 / 24 )) if [ $__threshold -le $__untilSeconds ]; then #printf "[OK] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" printf "[OK] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateSummary.txta elif [ $__remainingDays -le 0 ]; then #printf "[CRITICAL] ===> $__alias <=== !!! Certificate '$__alias' has already expired !!!\n" printf "[CRITICAL] ===> $__alias <=== !!! Certificate '$__alias' has already expired !!!\n" >> $__scriptPath/certificateSummary.txt else #printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateSummary.txt printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateExpireWarning.txt fi done #Decide on ALERT __lcCEW=$(cat $__scriptPath/certificateExpireWarning.txt | wc -l) if [ $__lcCEW -gt 0 ]; then (>&2 echo "!!! [WARNING] Check expired certificates !!!") (>&2 echo "$(cat $__scriptPath/certificateExpireWarning.txt)") #Comment out if you want to send as WARNING email. #cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo exit 1 else echo "Script executed successfully! Certificates are OK!" echo " " echo "##################################################" cat $__scriptPath/certificateSummary.txt exit 0 fi |
Share admineer resources...
Hi,
Thanks for the article. I have certain queries regarding how to edit environment variables and also how to create the cron job. Can I contact you. If yes, need your contact detail.
Please advise.
Hi,
Yes you can reach me by using contact 
These queries are common and off topic about using the script. Just google it
Hi,
I need one help with the script.
In my company environment there are multiple JVM’s running on single server each is having a separate JKS file. Is there a way to consolidate all the JKS file and get the result.
Eg:
__keystore=”/webdata/dev/PT855H92/webserv/HRGPSD1/piaconfig/keystore/pskey”
__keystore=”/webdata/sit/PT855H92/webserv/HRGPST1/piaconfig/keystore/pskey”
__keystore=”/webdata/dev/PT855H92/webserv/HRPSPD1/piaconfig/keystore/pskey”
__keystore=”/webdata/sit/PT855H92/webserv/HRPSPT1/piaconfig/keystore/pskey”
Do we have any script to get results from all the keystores at once. Instead of creating separate script for each JKS File.
Regards,
Raghu
Thank you for this script. However, my Alias name is different, On running this loop
for i in $($__keytool -list -v -keystore $__keystore -storepass $__keystorepass | grep ‘Alias name:’ | perl -ne ‘if(/name: (.*?)\n/) { print “$1\n”; }’)
===OUTPUT=====
verisignclass2g2ca [jdk]
digicertassuredidg3 [jdk]
verisignuniversalrootca [jdk]
digicerttrustedrootg4 [jdk]
You see the Alias name is not a single word as “verisignclass2g2ca” rather it is “verisignclass2g2ca [jdk]” which is failing in the second loop
do
echo “$i valid until: “$($__keytool -list -v -keystore $__keystore -storepass $__keystorepass -alias “$i” | grep ‘Valid from’ | head -1 | perl -ne ‘if(/until: (.*?)\n/) { print “$1\n”; }’) >> $__scriptPath/certificateStatus.txt
done
Because of $i is able to parse only verisignclass2g2ca or [jdk] as different Alias and the script fails
Do you have any idea, how we can treat “verisignclass2g2ca [jdk]” as one Alias name?
Regards,
Rohit
Hi Rohit,
You need to remove “[jdk]” part or escape the space, parentheses characters. You can put the complete output here, then I would have a look.
Hi,
Could you please provide a batch script to send alert based on number of days
The only thing is setup a cronjob then email output.
test comment
Hi Rohit,
You need to remove “[jdk]” part or escape the space, parentheses characters. You can put the complete output here, then I would have a look.