OK so I know which Zenpacks use these variables, in fact these two are not the only variables that fail when it starts to fail. There are more templates now using these variables than were in the original Zenpack, and the templates that call on these variables usually work using the context they are currently configured for, one can get the commands using the zendmd method of calling getDeviceCommands() for the devices that eventually fail to collect later on.
Although there were thresholds with complex tales expressions we have generally deleted them all while diagnosing this issue and well before the most recent occurence so that is not too relevant now.
Sample outputs from the greps suggested:
/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.RomanTykhonov.OracleDB-2.0-py2.6.egg/ZenPacks/RomanTykhonov/OracleDB/objects/objects.xml:${here/ZenPackManager/packs/ZenPacks/RomanTykhonov/OracleDB/objects/objects.xml:${here/ZenPackManager/packs/ZenPacks.RomanTykhonov.OracleDB/path}/libexec/oracle_status.sh "${device/zOracleUser}/${device/zOraclePass}@${device/zOracleDBInstance}" ${device/zOracleHome}
/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.RomanTykhonov.OracleDB-2.0-py2.6.egg/ZenPacks/RomanTykhonov/OracleDB/__init__.py: ('zOracleUser','sys', 'string'),
/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/datasources/JMXDataSource.py: authenticate = '${here/zJmxAuthenticate}'
/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/__init__.py: ('zJmxAuthenticate', False, 'boolean'),
Here is a typical usage of the variables in a JMX datasource. Note that the variable are typically referred to with "${here/", when is that not appropriate?
What does not make sense to me is why zenhub is happy to traverse and find an instance of the required variables even if not instantiated at the device level for days on end and then suddenly start having context issues for what seems like any variable which is not. zJmxAuthenticate for intance is typically set with a useful value at / so is not normally set down at the device level. Remember all ZenJMX and zencommand data sources that need a zProperty from a higher level than the device get dropped from every collector when this problem starts happening. I think we may be looking for one config or action that causes everything else to break or could it be a "try:" Python operation which fails to catch the exception eventually such as at line:
File "/usr/local/zenoss/zenoss/Products/ZenHub/services/CommandConfig.py", line 87, in getDeviceCommands
try:
threshs = getComponentCommands(dev, cache, cmds, dev.getDmd())
except (SystemExit, KeyboardInterrupt), ex:
log.exception("Unable to process device commands for %s -- skipping",
dev.id)
I notice that the Oracle templates from the RomanTykhonov Zenpack seem to all refer to their variables as "${device/" and sure enough we tend to fail on those variables like zOracleHome which are often literally the same for all devices so are not instantiated at the device level but instead at the device type level that the template is associated with which is /Server/Oracle as it comes from the Zenpack.
To add insult to injury, the system has been inexplicably stable with regards to this problem for 5 days, the longest period in months, though there have been myriad zenoss restarts in that time due to other needs that might have reset the clock/time bomb whatever that is.