Hello,
I have some new collectors I setup using the distribution collector zenpack on a new installation of zenoss 4.2.3. I have a collector that cannot seem to run zencommand or zenperfsnmp, ect against the server it is supposed to be collecting data from. When I run one of those commands:
zenperfsnmp run -d <server> -v10
2013-09-24 11:35:16,739 DEBUG zen.zenperfsnmp: Starting PBDaemon initialization
2013-09-24 11:35:16,740 INFO zen.zenperfsnmp: Connecting to <zenmaster>:8789
2013-09-24 11:35:16,744 DEBUG zen.pbclientfactory: Starting connection...
2013-09-24 11:35:16,744 DEBUG zen.zenperfsnmp: Logging in as admin
2013-09-24 11:35:16,746 DEBUG zen.pbclientfactory: Connected
2013-09-24 11:35:16,746 DEBUG zen.pbclientfactory: Cancelling connect timeout
2013-09-24 11:35:16,747 DEBUG zen.pbclientfactory: Sending credentials
2013-09-24 11:35:16,751 DEBUG zen.pbclientfactory: Cancelling connect timeout
2013-09-24 11:35:16,752 INFO zen.zenperfsnmp: Connected to ZenHub
2013-09-24 11:35:16,752 DEBUG zen.zenperfsnmp: Setting up initial services: EventService, Products.ZenHub.services.SnmpPerformanceConfig
2013-09-24 11:35:16,753 DEBUG zen.zenperfsnmp: Chaining getInitialServices with d2
2013-09-24 11:35:16,753 DEBUG zen.pbclientfactory: pinging perspective
2013-09-24 11:35:16,758 DEBUG zen.zenperfsnmp: Loaded service EventService from zenhub
2013-09-24 11:35:16,758 DEBUG zen.zenperfsnmp: Loaded service Products.ZenHub.services.SnmpPerformanceConfig from zenhub
2013-09-24 11:35:16,759 DEBUG zen.zenperfsnmp: Queued event (total of 1) {'rcvtime': 1380036916.758895, 'severity': 0, 'component': 'zenperfsnmp', 'agent': 'zenperfsnmp', 'summary': 'started', 'manager': '<collector>', 'device': 'localhost', 'eventClass': '/App/Start', 'monitor': 'localhost'}
2013-09-24 11:35:16,760 DEBUG zen.zenperfsnmp: Sending 1 events, 0 perf events, 0 heartbeats
2013-09-24 11:35:16,761 DEBUG zen.zenperfsnmp: Calling connected.
2013-09-24 11:35:16,761 DEBUG zen.collector.config: Heartbeat timeout set to 900s
2013-09-24 11:35:16,762 DEBUG zen.collector.scheduler: add task configLoader, <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8dd0> using 1200 second interval
2013-09-24 11:35:16,762 DEBUG zen.zenperfsnmp: Performing periodic maintenance
2013-09-24 11:35:16,762 DEBUG zen.pbclientfactory: perspective ponged
2013-09-24 11:35:16,763 DEBUG zen.pbclientfactory: Cancelling ping timeout
2013-09-24 11:35:16,763 DEBUG zen.collector.scheduler: Task configLoader starting on 1200 second intervals
2013-09-24 11:35:16,763 DEBUG zen.collector.scheduler: Task configLoader changing state from IDLE to QUEUED
2013-09-24 11:35:16,764 DEBUG zen.collector.scheduler: Task configLoader changing state from QUEUED to RUNNING
2013-09-24 11:35:16,764 DEBUG zen.collector.config: configLoader gathering configuration
2013-09-24 11:35:16,765 DEBUG zen.collector.config: Fetching daemon configuration properties
2013-09-24 11:35:16,782 DEBUG zen.collector.scheduler: Task configLoader changing state from RUNNING to FETCHING_MISC_CONFIG
2013-09-24 11:35:16,782 DEBUG zen.zenperfsnmp: Updated configCycleInterval preference to 360
2013-09-24 11:35:16,782 DEBUG zen.zenperfsnmp: Changing config task interval from 20 to 360 minutes
2013-09-24 11:35:16,782 DEBUG zen.collector.scheduler: Stopping task configLoader, <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8dd0>
2013-09-24 11:35:16,783 DEBUG zen.collector.scheduler: call finished LoopingCall<1200>(CallableTask: configLoader, *(), **{}) : LoopingCall<1200>(CallableTask: configLoader, *(), **{})
2013-09-24 11:35:16,783 INFO zen.collector.scheduler: Detailed Task Statistics:
configLoader Current State: FETCHING_MISC_CONFIG Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0
Detailed Task States:
configLoader State: RUNNING Total: 1 Total Elapsed: 0.0177 Min: 0.0177 Max: 0.0177 Mean: 0.0177 StdDev: 0.0000
configLoader State: QUEUED Total: 1 Total Elapsed: 0.0008 Min: 0.0008 Max: 0.0008 Mean: 0.0008 StdDev: 0.0000
2013-09-24 11:35:16,783 DEBUG zen.collector.config: Heartbeat timeout set to 900s
2013-09-24 11:35:16,784 DEBUG zen.collector.scheduler: add task configLoader, <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8e90> using 21600 second interval
2013-09-24 11:35:16,784 DEBUG zen.zenperfsnmp: Updated defaultRRDCreateCommand preference to ('RRA:AVERAGE:0.5:1:600', 'RRA:AVERAGE:0.5:6:600', 'RRA:AVERAGE:0.5:24:600', 'RRA:AVERAGE:0.5:288:600', 'RRA:MAX:0.5:6:600', 'RRA:MAX:0.5:24:600', 'RRA:MAX:0.5:288:600')
2013-09-24 11:35:16,785 DEBUG zen.collector.config: Fetching threshold classes
2013-09-24 11:35:16,794 DEBUG zen.zenperfsnmp: Loading classes ['Products.ZenModel.MinMaxThreshold', 'Products.ZenModel.ValueChangeThreshold']
2013-09-24 11:35:16,796 DEBUG zen.collector.config: Fetching collector thresholds
2013-09-24 11:35:16,850 DEBUG zen.thresholds: Updating threshold ('high event queue', ('localhost collector', ''))
2013-09-24 11:35:16,851 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('localhost collector', ''))
2013-09-24 11:35:16,851 DEBUG zen.collector.config: Fetching configurations
2013-09-24 11:35:17,010 ERROR zen.collector.config: Configuration for <server> unavailable -- is that the correct name?
2013-09-24 11:35:17,010 DEBUG zen.collector.scheduler: Task configLoader finished, result: 'Configuration loaded'
2013-09-24 11:35:17,011 DEBUG zen.collector.scheduler: In shutdown stage before
2013-09-24 11:35:17,011 DEBUG zen.collector.scheduler: Removing task configLoader
2013-09-24 11:35:17,011 DEBUG zen.collector.scheduler: tasks to clean set([<Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8e90>, <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8dd0>])
2013-09-24 11:35:17,011 DEBUG zen.collector.scheduler: Cleanup on task configLoader <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8e90>
2013-09-24 11:35:17,012 DEBUG zen.collector.scheduler: Scheduler._cleanupTaskComplete: result=None task.name=configLoader
2013-09-24 11:35:17,012 DEBUG zen.collector.scheduler: Cleanup on task configLoader <Products.ZenCollector.config.ConfigurationLoaderTask object at 0x46f8dd0>
2013-09-24 11:35:17,012 DEBUG zen.collector.scheduler: Scheduler._cleanupTaskComplete: result=None task.name=configLoader
2013-09-24 11:35:17,012 DEBUG zen.zenperfsnmp: Tried to stop reactor that was stopped
2013-09-24 11:35:17,013 INFO zen.zenperfsnmp: Daemon CollectorDaemon shutting down
2013-09-24 11:35:17,015 DEBUG zen.zenperfsnmp: Removing service EventService
2013-09-24 11:35:17,015 DEBUG zen.zenperfsnmp: Removing service Products.ZenHub.services.SnmpPerformanceConfig
2013-09-24 11:35:17,015 DEBUG zen.pbclientfactory: Lost connection to <zenmaster>:8789 - [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2013-09-24 11:35:17,015 DEBUG zen.collector.scheduler: In shutdown stage during
2013-09-24 11:35:17,016 DEBUG zen.collector.scheduler: In shutdown stage after
The following service run on my collector (note I have disabled some services on purpose)
zenoss status
Daemon: zenping program running; pid=18086
Daemon: zensyslog program running; pid=18167
Daemon: zenstatus program running; pid=18161
Daemon: zentrap program running; pid=18248
Daemon: zenmodeler program running; pid=18243
Daemon: zenperfsnmp program running; pid=18278
Daemon: zencommand program running; pid=18308
Daemon: zenprocess program running; pid=18330
Daemon: zenjmx program running; pid=18363
Daemon: zenrrdcached program running; pid=18399
Daemon: zenrender program running; pid=18442
I have tried to remove the collector and re-add it. I have triple check that all the configs and hub password files are correct. When I run a netstat it looks like all the proper ports are connected to the master (particularly 8789). There is no firewall between the master and collector node. I got my network security team to verify this for me. I have run 'zenchkrels -r -x1' on the master and no errors need repair. I have put most logs on the master and collector to debug but I don't see any errors pertaining to this problem. Thanks for any help!